How do scientific methods contribute to the development of modern science?

Why is NASA's cost modeling and estimation so off when analyzing SpaceX's development methodology in comparison to NASA's own?

• NASA estimated SpaceX should have spent $1.7-$4.0 bn to develop the Falcon 9 rocket - in reality, SpaceX spent $0.3-$0.4 bn. "NASA recently conducted a predicted cost estimate of the Falcon 9 launch vehicle using the NASA-Air Force Cost Model (NAFCOM).  NAFCOM is the primary cost estimating tool NASA uses to predict the costs for launch vehicles, crewed vehicles, planetary landers, rovers, and other flight hardware elements prior to the development of these systems.  NAFCOM is a parametric cost estimating tool with a historical database of over 130 NASA and Air Force space flight hardware projects. It has been developed and refined over the past 13 years with 10 releases providing increased accuracy, data content, and functionality. NAFCOM uses a number of technical inputs in the estimating process. These include mass of components, manufacturing methods, engineering management, test approach, integration complexity, and pre-development studies.  Another variable is the relationship between the Government and the contractor during development.  At one end, NAFCOM can model an approach that incorporates a heavy involvement on the part of the Government, which is a more traditional approach for unique development efforts with advanced technology.  At the other end, more commercial-like practices can be assumed for the cost estimate where the contractor has more responsibility during the development effort. For the Falcon 9 analysis, NASA used NAFCOM to predict the development cost for the Falcon 9 launch vehicle using two methodologies: 1) Cost to develop Falcon 9 using traditional NASA approach, and 2) Cost using a more commercial development approach. Under methodology #1, the cost model predicted that the Falcon 9 would cost $4.0 billion based on a traditional approach.  Under methodology #2, NAFCOM predicted $1.7 billion when the inputs were adjusted to a more commercial development approach.  Thus, the predicted the cost to develop the Falcon 9 if done by NASA would have been between $1.7 billion and $4.0 billion. SpaceX has publicly indicated that the development cost for Falcon 9 launch vehicle was approximately $300 million. Additionally, approximately $90 million was spent developing the Falcon 1 launch vehiclewhich did contribute to some extent to the Falcon 9, for a total of $390 million. NASA has verified these costs. It is difficult to determine exactly why the actual cost was so dramatically lower than the NAFCOM predictions.  It could be any number of factors associated with the non-traditional public-private partnership under which the Falcon 9 was developed (e.g., fewer NASA processes, reduced oversight, and less overhead), or other factors not directly tied to the development approach.  NASA is continuing to refine this analysis to better understand the differences.  Regardless of the specific factors, this analysis does indicate the potential for reducing space hardware development costs, given the appropriate conditions.  It is these conditions that NASA hopes to replicate, to the extent appropriate and feasible, in the development of commercial crew transportation systems." --pg. 40, http://www.nasa.gov/pdf/543572main_Section%20403%28b%29%20Commercial%20Market%20Assessment%20Report%20Final.pdf

• Answer:

  NAFCOM is a parametric cost estimating tool with a historical database of over 130 NASA and Air Force space flight hardware projects. It has been developed and refined over the past 13 years with 10 releases providing increased accuracy, data content, and functionality. 130 missions over 13 years is not a good sample size, especially when applied to new entrants. When one applies this metric, one assumes the entrant is like those who have come before. Furthermore, the vast majority of space flight hardware is made by either custom one-off shops or behemoths like Lockheed Martin. The former is more expensive due to the niche goods they create (essentially a monopoly), the latter due to their size, lobbying power and "connections" within the space industry. Secondly, a lot has changed over the last 13 years. Cheap computing allows the majority of the craft to be simulated before its first part is even fabricated, dramatically reducing the cost to market. In turn they can simulate multiple times, and when they are happy fabricate it and test it in real life. This was not easily done 10 years ago as computation was very expensive. Thirdly, SpaceX is a tiny startup with a small team that represents the best of the best in the aerospace industry. In turn their cost per rocket per employee is much lower, as there is less waste and more output per employee. Finally, SpaceX vertically integrates a very standard and modular design, which attempts to reduce reliance on third party niche contractors (expensive/unreliable). In turn their craft is simple to create, simple to manufacture, simple to assemble, and simple to operate (kerosene fuel instead of liquid hydrogen). All three things reduce cost to market, as you cut out the monopoly contractors, use commodity goods, use simple designs to reduce error/flaws/maintenance and make it modular so that every part is easily replaceable.