What is Facebook IPO?

What was the software design flaw in Nasdaq application that caused the Facebook IPO debacle?

• "Brokers handling http://www.bloomberg.com/quote/FB:US orders in the May 2012 IPO claimed they lost hundreds of millions of dollars after a design flaw in Nasdaq’s software delayed the stock’s open and left them confused about whether or not they owned shares." What was the software design flaw in Nasdaq application that caused the Facebook IPO debacle? http://www.bloomberg.com/news/2013-05-29/nasdaq-agrees-to-pay-10-million-to-settle-sec-facebook-charges.html

• Answer:

  There were really a few different software bugs detailed in the SEC report. Some were related to design, others were the result of human intervention, and others were oversights in contradiction of NASDAQ policies. The software-specific issues were primarily related to the NASDAQ Cross and Execution Applications, which are described in more detail in the formal SEC report: http://www.sec.gov/litigation/admin/2013/34-69655.pdf At the conclusion of the Display Only Period there was a 10 second delay prior to execution of the cross. This delay is caused by an intentional randomization factor in many NASDAQ operations, where a delay from 1-15 seconds is randomly chosen prior to performing an action. Although NASDAQ had previously stated they were removing these randomization elements from their execution platform, they had never actually been removed and were still in effect. When the NASDAQ Cross Application attempted to reconcile orders received during the DOP and before secondary trading formally began it encountered a race condition. Typically execution of the cross takes 1-2 milliseconds but due to the high volume of the Facebook IPO in this case it took 20+ milliseconds. During this time additional cancel orders were received, which caused the Cross App to "fail" and restart the cross computation, incurring another 20+ms delay. During each of these delays additional cancellation orders were received, which caused the Cross App to continually restart each time it completed the cross. Also, instead of processing the cancelled orders as a batch, the Cross App would serially apply each cancellation to its Cross run, such that one cancel would cause one re-cross, while four cancels would trigger four crosses, performed serially. The specific bug the question details refers to was a result of NASDAQ's failover to standby trading systems, but not reconciling 19 minutes of order cancellation data during the cutover. The SEC report details the discovery and ramifications of this error: When the bulk print was released, certain NASDAQ executives, including its Senior Vice President and Chief Economist (“Chief Economist”), noticed the approximately 6.3 million share difference between the final indicative volume total (82 million) and the actual volume in the print (75.7 million). This discrepancy indicated to NASDAQ’s Chief Economist that there was still a problem with the cross, and that some cross eligible orders may not have been handled properly in the cross but NASDAQ did not address this issue during the minutes and hours following the cross. In addition, NASDAQ could have run a real-time status check of its applications, which would have indicated that the cross executed at 11:30 a.m. did not include any orders entered after 11:11 a.m. The fact that the IPO Cross Application failed to include 19 minutes of orders in its price/volume calculation also substantially impacted the error position that NASDAQ assumed based on its decision at 11:25 a.m. to eliminate the validation check when it moved to the failover system. Instead of assuming a position covering an imbalance created during the expected fraction of a second between the start and the completion of the cross calculation,NASDAQ ended up assuming a position resulting from the imbalance between buy cancels and sell cancels entered between 11:11 a.m. and 11:30:09 a.m. Because more sell shares than buy shares were cancelled during this period, NASDAQ had a more than 3 million share short position in Facebook that it needed to cover in order to fill the excess buy orders. An additional software issue was uncovered after trading commenced, when NASDAQ's "Execution Application" failed to receive accurate data about the cross, which resulted in NASDAQ publishing bad pricing data to the market. Typically the Cross Application publishes a final bundle of data to the Execution App about a bulk print prior to the start of secondary trading, but because NASDAQ intentionally bypassed the Cross App to start trading, this data was never delivered. The Execution App's bad data caused trade confirmations to not be delivered to NASDAQ members (the large Wall Street banks performing the trades), which further exacerbated the lack of data and uncertainty in the market, which depressed trading activity. Although some NASDAQ data feeds had more accurate data, it required people knowledgeable with the various data sources to manually filter known-bad data to understand the actual state of Facebook's stock once it began trading. In addition to all of the above problems, 's stock was also affected in part due to its high reliance on Facebook. When Zynga's stock dropped by 10% it triggered an automatic market halt for their stock. However, market halts are handled by the same system that performs IPO crosses, which was already in a degraded state due to the earlier Facebook issues. After the halt was lifted there was a set of trades that were improperly executed due to the Cross App issues. Zynga went through another cycle of halting trades after resuming, due to another 10% drop in their stock price.