One of the biggest sources of pain in system development is "system integration and test." This is frequently where projects sailing along with all-green progress reports and Earned Value Management System status summaries start to see these indicators increasingly turn to yellow and then to red. Projects that were thought to be 80 percent complete may be found to still have another 120 percent to go, increasing the relative costs of integration and test from 20 percent of the total to 120/200 = 60 percent of the total.

Managers often look at this 60 percent figure and say, "We need to find a way to speed up integration and test," and invest in test tools to make testing go faster. But this is not the root cause of the cost escalation. That happened a lot earlier in the definition and validation (or more often the lack of these) of the system's architecture. Components that were supposed to fit together did not. Unsuspected features in commercial off-the-shelf (COTS) products were found to be incompatible, with no way to fix them and little vendor interest in doing anything about the problems. Nominal-case tests worked beautifully but the more frequent off-nominal cases led to system failures. Readiness tests for safety and security certification were unacceptable. Defect fixes caused regression tests to fail due to unanticipated side effects. Required response times were impossible to meet. And award fees for on-time delivery and expected career promotions faded away.

.