GAO Report to the Honorable Edward J. Markey, House of Representatives February 2002 MISSILE DEFENSE Review of Results and Limitations of an Early National Missile Defense Flight Test GAO-02-124 Contents Letter Disclosure of Key Results and Limitations Project Office Reliance on Various Sources for Contractor Oversight Distinguishable Differences in Objects Deployed in Space Decoy Reduction in Later Tests Evaluation of TRW’s Discrimination Software Agency Comments and Our Evaluation 1 5 7 7 8 8 9 Appendix I Disclosure of Flight Test’s Key Results and Limitations 11 The Test 11 Reported Key Results and Limitations 13 Effect of Cooling Failure on Sensor’s Performance 22 Appendix II Project Office Reliance on Various Sources for Contractor Oversight Appendix III Reduced Test Complexity 26 Decoys in Early Intercept Tests 26 Opinions on Decoys 28 Appendix IV Phase One Engineering Team’s Evaluation of TRW’s Software 30 Phase One Engineering Team’s Methodology 30 The Phase One Engineering Team’s Key Results 32 Limitations of the Team’s Evaluation 33 Appendix V Boeing Integrated Flight Test 1A Requirements and Actual Performance as Reported by Boeing and TRW 34 Appendix VI Scope and Methodology 36 Appendix VII Comments from the Department of Defense 38 Appendix VIII Major Contributors 39 Acquisition and Sourcing Management 39 Applied Research and Methods 39 General Counsel 39 Tables Table 1: What and When Key Results and Limitations Were Included in Contractors’ Written Reports 13 Table 2: Planned and Actual Targets for Initial Flight Tests 27 Table 3: Integrated Flight Test 1A Requirements Established by Boeing and Actual Performance 34 United States General Accounting Office Washington, DC 20548 February 28, 2002 The Honorable Edward J. Markey House of Representatives For a number of years, the Department of Defense has been researching and developing defenses against ballistic missile attacks on the United States, its deployed forces, friends, and allies. In 1990, the Department awarded research and development contracts to three contractors to develop and test exoatmospheric kill vehicles.1 The Department planned to use the best of the three vehicles in a follow-on missile defense program. One of the contractors, Rockwell International, subcontracted a portion of its kill vehicle design work to TRW. TRW was tasked with developing software that could operate on a computer onboard the kill vehicle. The software was to analyze data collected in flight by the kill vehicle’s sensor (which collects real-time information about threat objects), enabling the kill vehicle to distinguish an enemy warhead from accompanying decoys.2 The three contractors proceeded with development of the kill vehicle designs and built and tested key subsystems (such as the sensor) until 1994. In 1994, the Department of Defense eliminated Martin Marietta from the competition. Both Rockwell—portions of which in December 1996 became Boeing North American—and Hughes—now Raytheon— continued designing and testing their kill vehicles. In 1997 and 1998, the National Missile Defense Joint Program Office3 conducted tests, in space, of the sensors being developed by the contractors for their competing kill vehicles. Boeing's sensor was tested in June 1997 (Integrated Flight Test 1A) and Raytheon's sensor was tested in January 1998 (Integrated Flight Test 2). Program officials said these tests were not meant to demonstrate that the sensor met performance requirements, nor were they intended to be the basis for any contract award decisions. Rather, they were early research and development tests that the program office considered 1 An exoatmospheric kill vehicle is the part of a defensive missile that is designed to hit and destroy an incoming enemy warhead above the earth’s atmosphere. 2In some instances, the system may also use ground radar data. 3 The National Missile Defense Joint Program Office reports to the Ballistic Missile Defense Organization within the Department of Defense. The National Missile Defense program is now known as the Ground-based Midcourse Missile Defense Program and the Ballistic Missile Defense Organization is now the Missile Defense Agency. experiments to primarily reduce risk in future flight tests. Specifically, the tests were designed to determine if the sensor could operate in space; to examine the extent to which the sensor could detect small differences in infrared emissions; to determine if the sensor was accurately calibrated; and to collect target signature4 data for post-mission discrimination analysis. After the two sensor tests, the program office planned another 19 flight tests from 1999 through 2005 in which the kill vehicle would attempt to intercept a mock warhead. Initially, Boeing’s kill vehicle was scheduled for testing in Integrated Flight Test 3 and Raytheon’s in Integrated Flight Test 4. However, Boeing became the Lead System Integrator for the National Missile Defense Program in April 1998 and, before the third flight test was conducted, selected Raytheon as the primary kill vehicle developer.5 Meanwhile, in September 1995, TRW had hired a senior staff engineer, Dr. Nira Schwartz, to work on various projects, including the company’s effort to develop the exoatmospheric kill vehicle’s discrimination software. The engineer helped evaluate some facets of a technology known as the Extended Kalman Filter Feature Extractor,6 which TRW planned to add as an enhancement to its discrimination software. The engineer reported to TRW in February 1996 that tests revealed that the Filter could not extract the key characteristics, or features, from various target objects that an enemy missile might deploy and demanded that the company inform Rockwell and the Department of Defense. TRW fired the engineer in March 1996. In April 1996, the engineer filed a lawsuit under the False Claims Act7 alleging that TRW8 falsely reported or hid information to make the National Missile Defense Joint Program Office believe that the Extended Kalman Filter Feature Extractor met the 4 A target object’s signature is the set of infrared signals emitted by the target. 5 The Department of Defense continued funding the Boeing kill vehicle at a reduced level as a backup to Raytheon’s kill vehicle. In mid-2000, the Department terminated all funding for Boeing’s kill vehicle, ending TRW’s involvement in development of the kill vehicle’s discrimination software. 6 The Kalman Filter is a mathematical model commonly used in real time data processing to estimate a variable of interest, such as an object’s position or velocity. The Extended Kalman Filter Feature Extractor is used to extract features, which are used to perform discrimination. 7 31 USC 3729-3733. 8 Rockwell, now Boeing North American, was later added to the lawsuit. Department’s technical requirements. The engineer has amended the lawsuit several times, including adding allegations that TRW misled the Department of Defense about the ability of its discrimination software to distinguish a warhead from decoys and that TRW's test reports on Integrated Flight Test 1A falsely represented the discrimination software’s performance. The False Claims Act allows a person to bring a lawsuit on behalf of the U.S. government if he or she has knowledge that a person or company has made a false or fraudulent claim against the government. If the suit is successful, the person bringing the lawsuit may share in any money recovered. The Department of Justice reviews all lawsuits filed under the act before deciding whether to join them. If it does, it becomes primarily responsible for prosecuting the case. To determine whether it should join the engineer's lawsuit against TRW, Justice asked the Defense Criminal Investigative Service, a unit within the Department of Defense Inspector General’s office,9 to examine the allegations. The engineer cooperated with the Investigative Service for more than 2 years. During the course of the Department of Defense’s investigation into the allegations of contractor fraud, two groups examined the former employee’s specific allegations regarding the performance of TRW’s basic discrimination software and performed limited evaluations of the Extended Kalman Filter Feature Extractor. The first was Nichols Research Corporation, a contractor providing technical assistance to the Ground Based Interceptor Project Management Office for its oversight of the exoatmospheric kill vehicle contracts. (This office within the National Missile Defense Joint Program Office is responsible for the exoatmospheric kill vehicle contracts.) Because an investigator for the Defense Criminal Investigative Service was concerned about the ability of Nichols to provide a truly objective assessment, the National Missile Defense Joint Program Office asked an existing advisory group, known as 9 Department of Justice officials told us that they often use other agencies’ investigative units to investigate contractor fraud cases. the Phase One Engineering Team,10 to undertake another review of the specific allegations of fraud with respect to the software. This group is comprised of scientists from Federally Funded Research and Development Centers who were selected for the review team because of their knowledge of the National Missile Defense system. In addition, both Nichols and the Phase One Engineering Team assessed the feasibility of using the Extended Kalman Filter Feature Extractor to extract additional features from target objects that an enemy missile might deploy.11 The Department of Justice and the Defense Criminal Investigative Service investigated the engineer’s allegations until March 1999. At that time, the Department of Justice decided not to intervene in the lawsuit. The engineer has continued to pursue her lawsuit without Justice’s intervention. When a Massachusetts Institute of Technology professor, Dr. Theodore Postol, learned of the engineer’s claims, he conducted his own analysis of Integrated Flight Test 1A. In May 2000, the professor wrote to the White House alleging that Boeing North American and TRW misrepresented the results of the test. The professor claimed that his analysis of Integrated Flight Test 1A showed that the system can be defeated by the simplest of decoys and that the National Missile Defense Joint Program Office and its contractors attempted to hide this fact by tampering with the flight test data and altering their analysis of the sensor’s discrimination capabilities. The professor also alleged that objects deployed as part of Integrated Flight Test 1A displayed no distinguishable differences that Boeing’s infrared 10 The Phase One Engineering Team, according to its director, was established in 1988 by the Strategic Defense Initiative Organization—later known as the Ballistic Missile Defense Organization—as an umbrella mechanism to obtain technical and engineering support from Federally Funded Research and Development Centers. To ensure that the scientists who work on each review undertaken by the Phase One Engineering Team have the requisite expertise in the subjects they are asked to review, the membership on each review team varies with each assignment. The team assembled to review TRW’s software included two individuals from the Massachusetts Institute of Technology’s Lincoln Laboratory, two from Lawrence Livermore National Laboratory, and one from the Aerospace Corporation. 11 In October 1996, TRW removed the Extended Kalman Filter Feature Extractor from its discrimination software. According to company officials, the Filter required computer speed and memory resources that were not available in the kill vehicle’s onboard processor. In addition, the officials said that the basic discrimination software would perform adequately even without the Filter. sensor could use to identify the mock warhead from decoys and that the program office hid the sensor’s weaknesses by reducing the number of decoys planned for future tests. Further, the professor claimed that the Phase One Engineering Team’s analysis was faulty. At your request, we reviewed the professor’s allegations. Specifically, as discussed with your office, we addressed the following questions: 1. Did Boeing and TRW disclose the key results and limitations of the flight test to the National Missile Defense Joint Program Office? 2. How did the Ground Based Interceptor Project Management Office oversee Boeing’s and TRW’s technical performance? 3. Did the flight test show whether each object deployed in space by an attacking missile exhibits distinguishable features? 4. Why did the National Missile Defense Joint Program Office reduce the complexity of later flight tests? 5. What were the methodology, findings, and limitations of the evaluation conducted by the Phase One Engineering Team of TRW’s discrimination software? You also asked us to determine whether the Department of Defense misused the security classification process to stifle public discussion of possible problems with the National Missile Defense system. We addressed this question in a separate report, dated June 12, 2001.12 Boeing and TRW disclosed the key results and limitations of Integrated Flight Test 1A in written reports released between August 13, 1997, and April 1, 1998. The contractors explained in a report issued 60 days after the June 1997 test that the test achieved its primary objectives, but that some sensor abnormalities were noted.13 For example, while the report explained that the sensor detected the deployed targets and collected Disclosure of Key Results and Limitations 12 DOD Officials Acted in Accordance With Executive Order for Addressing Security Classification Concerns (GAO-01-737R, June 12, 2001). 13 Appendix V includes selected requirements that Boeing established before the flight test to evaluate sensor performance and the actual sensor performance characteristics that Boeing and TRW discussed in the report. some usable target signals, the report also stated that some sensor components did not operate as desired and the sensor often detected targets where there were none. In December 1997, the contractors documented other test anomalies. According to briefing charts prepared for a December meeting, the Boeing sensor tested in Integrated Flight Test 1A had a low probability of detection; the sensor’s software was not always confident that it had correctly identified some target objects; the software significantly increased the rank of one target object toward the end of the flight; and in-flight calibration of the sensor was inconsistent. Additionally, on April 1, 1998, the contractors submitted an addendum to an earlier report that noted two more problems. In this addendum, the contractors disclosed that their claim that TRW’s software successfully distinguished a mock warhead from decoys during a post-flight analysis was based on tests of the software using about one-third of the target signals collected during Integrated Flight Test 1A. The contractors also noted that TRW reduced the software’s reference data14 so that it would correspond to the collected target signals being analyzed. Project office and Nichols Research officials said that in late August 1997, the contractors orally communicated to them all problems and limitations that were subsequently described in the December 1997 briefing and the April 1998 addendum. However, neither project officials nor contractors could provide us with documentation of these communications. Although the contractors reported the test’s key results and limitations, they described the results using some terms that were not defined. For example, one written report characterized the test as a “success” and the sensor’s performance as “excellent.” We found that the information in the contractors’ reports, in total, enabled officials in the Ground Based Interceptor Project Management Office and Nichols Research to understand the key results and limitations of the test. However, because such terms are qualitative and subjective rather than quantitative and objective, their use increased the likelihood that test results would be interpreted in different ways and might even be misunderstood. As part of our ongoing review of missile defense testing, we are examining the need for improvements in test reporting. Appendix I provides details on the test and the information disclosed. 14 Reference data are a collection of predicted characteristics, or features, that target objects are expected to display during flight. The software identifies the warhead from the decoys by comparing the features displayed by the different target objects to the reference data. Project Office Reliance on Various Sources for Contractor Oversight The Ground Based Interceptor Project Management Office relied on an onsite engineer and Nichols Research Corporation to provide insight into Boeing’s work. The project office also relied on Boeing to oversee the performance of its subcontractor, TRW. Oversight was limited by the ongoing competition between Boeing and another contractor competing for the exoatmospheric kill vehicle contract because the Ground Based Interceptor Project Management Office and its support contractors had to be careful not to affect competition by assisting one contractor more than another. Project officials said that they relied more on “insight” into the contractors’ work rather than oversight of that work. Nichols gained program insight by attending technical meetings, assessing test reports, and sometimes evaluating technologies proposed by Boeing and TRW. For more information on how the project office exercised oversight over its contractors’ technical performance, see appendix II. Distinguishable Differences in Objects Deployed in Space Boeing and TRW reported that post-flight testing and analysis of data collected during Integrated Flight Test 1A showed that deployed target objects displayed distinguishable features when observed by an infrared sensor. The contractors reported the test also showed that Boeing’s exoatmospheric kill vehicle sensor could collect target signals from which TRW’s software could extract distinguishable features and that the software could identify the mock warhead from other objects by comparing the extracted features to the features that it had been told to expect each object to display. However, there has been no independent verification of these claims. We talked with Dr. Mike Munn, who was, during the 1980s, the Chief Scientist for missile defense programs at Lockheed Missiles and Space Company. He agreed that a warhead and decoys deployed in the exoatmosphere likely display distinguishable differences in the infrared spectrum. However, the differences may not be fully understood or there may not presently be methods to predict the differences. Dr. Munn added that the key was in the ability to make both accurate and precise measurements and also to predict signatures accurately. He emphasized that robust discrimination depends on the ability to predict signatures and then to match in-space measurements with those predictions. The Phase One Engineering Team and Nichols Research Corporation have noted that TRW's software used prior knowledge of warhead and decoy differences, to the maximum extent available, to discriminate one object from the other and cautioned such knowledge may not always be available in the real world. Decoy Reduction in Later Tests National Missile Defense program officials said that after considerable debate among themselves and contractors, the program manager reduced the number of decoys planned for intercept flight tests in response to a recommendation by an independent panel, known as the Welch Panel.15 The panel, established to reduce risk in ballistic missile defense flight test programs, viewed a successful hit-to-kill engagement as a difficult task that should not be further complicated in early tests by the addition of decoys. After contemplating the advice of the Welch panel and considering the opinions of program officials and contractors who disagreed over the number and complexity of decoys that should be deployed in future tests, the program manager decided that early tests should include only one decoy, a large balloon. See appendix III for more information on the reduction of decoys in later tests. Evaluation of TRW’s Discrimination Software The Phase One Engineering Team was tasked by the National Missile Defense Joint Program Office to assess the performance of TRW’s software and to complete the assessment within 2 months using available data. The team's methodology included determining if TRW’s software was based on sound mathematical, engineering, and scientific principles and testing the software’s critical modules using data from Integrated Flight Test 1A. The team reported that although the software had weaknesses, it was well designed and worked properly, with only some changes needed to increase the robustness of the discrimination function. Further, the team reported that the results of its test of the software using Integrated Flight Test 1A data produced essentially the same results as those reported by TRW. Based on its analysis, team members predicted that the software would perform successfully in a future intercept test if target objects deployed as expected. Because the Phase One Engineering Team did not process the raw data from Integrated Flight Test 1A or develop its own reference data, the team cannot be said to have definitively proved or disproved TRW’s claim that 15 The Welch Panel was chaired by Larry Welch, President of the Institute for Defense Analyses, and included 15 other members, some of whom were retired flag officers and former Department of Defense officials. its software successfully discriminated the mock warhead from decoys using data collected from Integrated Flight Test 1A. A team member told us its use of Boeing- and TRW-provided data was appropriate because the former TRW employee had not alleged that the contractors tampered with the raw test data or used inappropriate reference data. Appendix IV provides additional details on the Phase One Engineering Team evaluation. In commenting on a draft of this report, the Department of Defense concurred with our findings. It also suggested technical changes, which we incorporated as appropriate. The Department's comments are reprinted in appendix VII. Agency Comments and Our Evaluation We conducted our review from August 2000 through February 2002 in accordance with generally accepted government auditing standards. Appendix VI provides details on our scope and methodology. The National Missile Defense Joint Program Office’s process for releasing documents significantly slowed our work. For example, the program office took approximately 4 months to release key documents such as the Phase One Engineering Team’s response to the professor’s allegations. We requested these and other documents on September 14, 2000, and received them on January 9, 2001. As arranged with your staff, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we plan to provide copies of this report to the Chairmen and Ranking Minority Members of the Senate Committee on Armed Services; the Senate Committee on Appropriations, Subcommittee on Defense; the House Committee on Armed Services; and the House Committee on Appropriations, Subcommittee on Defense; and the Secretary of Defense; and the Director, Missile Defense Agency. We will make copies available to others upon request. If you or your staff have any questions concerning this report, please contact Bob Levin, Director, Acquisition and Sourcing Management, on (202) 512-4841; Jack Brock, Managing Director, on (202) 512-4841; or Keith Rhodes, Chief Technologist, on (202) 512-6412. Major contributors to this report are listed in appendix VIII. Sincerely yours, Jack L. Brock, Jr. Managing Director Acquisition and Sourcing Management Keith Rhodes Chief Technologist Applied Research and Methods Appendix I: Disclosure of Flight Test’s Key Results and Limitations Boeing and TRW disclosed the key results and limitations of an early sensor flight test, known as Integrated Flight Test 1A, to the Ground Based Interceptor Project Management Office. The contractors included some key results and limitations in written reports submitted soon after the June 1997 test, but others were not included in written reports until December 1997 or April 1998. However, according to project office and Nichols officials, all problems and limitations included in the written reports were communicated orally to the project management office in late August 1997. The deputy project office manager said his office did not report these verbal communications to others within the Program Office or the Department of Defense because the project office was the office within the Department responsible for the Boeing contract. One problem that was included in initial reports to program officials was a malfunctioning cooling mechanism that did not lower the sensor’s temperature to the desired level. Boeing characterized the mechanism’s performance as somewhat below expectations but functioning well enough for the sensor’s operation. We hired experts to determine the extent to which the problem could affect the sensor’s performance. The experts found that the cooling problem degraded the sensor’s performance in a number of ways, but would not likely result in extreme performance degradation. The experts studied only how increased noise1 affected the sensor’s performance regarding comparative strengths of the target signals and the noise (signal to noise ratio). The experts did not evaluate discrimination performance, which is dependent on the measurement accuracy of the collected infrared signals. The experts’ findings are discussed in more detail later in this appendix. Integrated Flight Test 1A, conducted in June 1997, was a test of the Boeing sensor—a highly sensitive, compact, infrared device, consisting of an array of silicon detectors, that is normally mounted on the exoatmospheric kill vehicle. However, in this test, a surrogate launch vehicle carried the sensor above the earth’s atmosphere to view a cluster of target objects that included a mock warhead and various decoys. When the sensor detected the target cluster, its silicon detectors began to make precise measurements of the infrared radiation emitted by the target objects. Over the tens of seconds that the target objects were within its field of view, the sensor continuously converted the infrared radiation into an electrical The Test 1 Noise is undesirable electronic energy from sources other than the target objects. Appendix I: Disclosure of Flight Test’s Key Results and Limitations current, or signal, proportional to the amount of energy collected by the detectors. The sensor then digitized the signal (converted the signals into numerical values), completed a preliminary part of the planned signal processing, and formatted the signal so that it could be transmitted via a data link to a recorder on the ground. After the test, Boeing processed the signals further2 and formatted them so that TRW could input the signals into its discrimination software to assess its capability to distinguish the mock warhead from decoys. In post-flight ground testing, the software analyzed the processed data and identified the key characteristics, or features, of each signal. The software then compared the features it extracted to the expected features of various types of target objects. Based on this comparison, the software ranked each item according to its likelihood of being the mock warhead. TRW reported that the highestranked object was the mock warhead. The primary objective of Integrated Flight Test 1A was to reduce risk in future flight tests. Specifically, the test was designed to determine if the sensor could operate in space; to examine the extent to which the sensor could detect small differences in infrared emissions; to determine if the sensor was accurately calibrated; and to collect target signature3 data for post-mission discrimination analysis. In addition, Boeing established quantitative requirements for the test.4 For example, the sensor was expected to acquire the target objects at a specified distance. According to a Nichols’ engineer, Boeing established these requirements to ensure that its exoatmospheric kill vehicle, when fully developed, could destroy a warhead with the single shot precision (expressed as a probability) required by the Ground Based Interceptor Project Management Office. The engineer said that in Integrated Flight Test 1A, Boeing planned to measure its sensor’s performance against these lower-level requirements so that Boeing engineers could determine which sensor elements, including the software, required further refinement. However, the engineer told us that because of the various sensor problems, of which the contractor and project office were aware, Boeing determined before the test that it would not use most of these requirements to judge the sensor’s performance. (Although Boeing did not judge the performance of its sensor against the 2 The signal processing that Boeing completed after the test will be completed onboard the exoatmospheric kill vehicle in an operational system. 3 A target object’s signature is the set of infrared signals emitted by the target. 4 These requirements were established by the contractor and were not imposed by the government. Appendix I: Disclosure of Flight Test’s Key Results and Limitations requirements as it originally planned, Boeing did, in some cases, report the sensor’s performance in terms of these requirements. For a summary of selected test requirements and the sensor’s performance as reported by Boeing and TRW in their August 22, 1997, report, see app. V.) Reported Key Results Table 1 provides details on the key results and limitations of Integrated Flight Test 1A that contractors disclosed in various written reports and and Limitations briefing charts. Table 1: What and When Key Results and Limitations Were Included in Contractors’ Written Reports August 13, 1997, Report August 22, 1997, Report December 11, 1997, Briefing April 1, 1998, Report Detected deployed targets Detected deployed targets High false alarm rate Failure of gap-filling modulea Target signals collected Target signals collected Sensor did not cool to desired Target signals collected during temperature selected portion of the flight timeline used in assessment of discrimination software Discrimination software distinguished mock warhead from decoys Discrimination software distinguished mock warhead from decoys Software confidence factor remained small for two target objects Selected reference data used in assessment of discrimination software Excellent performance of sensor payload Sensor had a lower than expected probability of detection Power supply caused noisy target signals Software significantly increased rank of one target object toward the end of the flight Sensor did not cool to desired temperature In-flight calibration of sensor was inconsistent High false alarm rate Slow turn-around of launch vehicle caused data loss aTRW designed a gap-filling module for its discrimination software to replace missing or noisy portions of collected and simulated target signals. Although the contractors disclosed the key results and limitations of the flight test in written reports and in discussions, the written reports described the results using some terms that were not defined. For example, in their August 22, 1997, report, Boeing and TRW described Integrated Flight Test 1A as a “success” and the performance of the Boeing sensor as “excellent.” We asked the contractors to explain their use of these terms. We asked Boeing, for example, why it characterized its sensor’s performance as “excellent” when the sensor’s silicon detector array did not cool to the desired temperature, the sensor’s power supply created excess noise, and the sensor detected numerous false targets. Boeing said that even though the silicon detector array operated at temperatures 20 to 30 percent higher than desired, the sensor produced Appendix I: Disclosure of Flight Test’s Key Results and Limitations useful data. Officials said they knew of no other sensor that would be capable of producing any useful data under those conditions. Boeing officials went on to say that the sensor continuously produced usable, and, much of the time, excellent data in “real-time” during flight. In addition, officials said the sensor component responsible for suppressing background noise in the silicon detector array performed perfectly in space and the silicon detectors collected data in more than one wave band. Boeing concluded that the sensor’s performance allowed the test to meet all mission objectives. Based on our review of the reports and discussions with officials in the Ground Based Interceptor Project Management Office and Nichols Research, we found that the contractors’ reports, in total, contained information for those officials to understand the key results and limitations of the test. However, because terms such as “success” and “excellent” are qualitative and subjective rather than quantitative and objective, we believe their use increases the likelihood that test results would be interpreted in different ways and could even be misunderstood. As part of our ongoing review of missile defense testing, we are examining the need for improvements in test reporting. The August 13 Report This report, sometimes referred to as the 45-day report, was a series of briefing charts. In it, contractors reported that Integrated Flight Test 1A achieved its principal objectives of reducing risks for subsequent flight tests, demonstrating the performance of the exoatmospheric kill vehicle’s sensor, and collecting target signature data. In addition, the report stated that TRW’s software successfully distinguished a mock warhead from accompanying decoys.5 The August 22 Report The August 22 report, known as the 60-day report, was a lengthy document that disclosed much more than the August 13 report. As discussed in more detail below, the report explained that some sensor abnormalities were observed during the test, that some signals collected from the target objects were degraded, that the launch vehicle carrying the sensor into 5 Boeing and TRW reported that the original test objectives did not include a test of TRW’s discrimination software. However, program officials decided immediately prior to the test that it offered an excellent opportunity to assess the software’s capability even though post-processing tools needed to assess the software were not yet available and would need rapid development after Integrated Flight Test 1A. Appendix I: Disclosure of Flight Test’s Key Results and Limitations Some Sensor Abnormalities Were Observed During the Test space adversely affected the sensor’s ability to collect target signals, and that the sensor sometimes detected targets where there were none. These problems were all noted in the body of the report, but the report summary stated that review and analysis subsequent to the test confirmed the “excellent” performance and nominal operation of all sensor subsystems. Boeing disclosed in the report that sensor abnormalities were observed during the test and that the sensor experienced a higher than expected false alarm rate. These abnormalities were (1) a cooling mechanism that did not bring the sensor’s silicon detectors to the intended operating temperature, (2) a power supply unit6 that created excess noise, and (3) software that did not function as designed because of the slow turnaround of the surrogate launch vehicle. In the report’s summary, Boeing characterized the cooling mechanism’s performance as somewhat below expectations but functioning well enough for the sensor’s operation. In the body of the report, Boeing said that the fluctuations in temperature could lead to an apparent decrease in sensor performance. Additionally, Boeing engineers told us that the cooling mechanism’s failure to bring the silicon detector array to the required temperature caused the detectors to be noisy. Because the discrimination software identifies objects as a warhead or a decoy by comparing the features of a target’s signal with those it expects a warhead or decoy to display, a noisy signal may confuse the software. Boeing and TRW engineers said that they and program office officials were aware that there was a problem with the sensor’s cooling mechanism before the test was conducted. However, Boeing believed that the sensor would perform adequately at higher temperatures. According to contractor documents, the sensor did not perform as well as expected, and some target signals were degraded more than anticipated. Boeing disclosed in the report that sensor abnormalities were observed during the test and that the sensor experienced a higher than expected false alarm rate. These abnormalities were (1) a cooling mechanism that did not bring the sensor’s silicon detectors to the intended operating temperature, (2) a power supply unit that created excess noise, and (3) software that did not function as designed because of the slow turnaround of the surrogate launch vehicle. In the report’s summary, Boeing characterized the cooling mechanism’s performance as somewhat below expectations but functioning well 6 The power supply unit is designed to power the sensor’s electronic components. Appendix I: Disclosure of Flight Test’s Key Results and Limitations Power Supply Creates Noise Payload Launch Vehicle Affected Software’s Ability to Remove Background Noise Sensor Sometimes Detected False Targets enough for the sensor’s operation. In the body of the report, Boeing said that the fluctuations in temperature could lead to an apparent decrease in sensor performance. Additionally, Boeing engineers told us that the cooling mechanism’s failure to bring the silicon detector array to the required temperature caused the detectors to be noisy. Because the discrimination software identifies objects as a warhead or a decoy by comparing the features of a target’s signal with those it expects a warhead or decoy to display, a noisy signal may confuse the software. Boeing and TRW engineers said that they and program office officials were aware that there was a problem with the sensor’s cooling mechanism before the test was conducted. However, Boeing believed that the sensor would perform adequately at higher temperatures. According to contractor documents, the sensor did not perform as well as expected, and some target signals were degraded more than anticipated. The report also referred to a problem with the sensor’s power supply unit and its effect on target signals. An expert we hired to evaluate the sensor’s performance at higher than expected temperatures found that the power supply, rather than the temperature, was the primary cause of excess noise early in the sensor’s flight. Boeing engineers told us that they were aware that the power supply was noisy before the test, but, as shown by the test, it was worse than expected. The report explained that, as expected before the flight, the slow turnaround of the massive launch vehicle on which the sensor was mounted in Integrated Flight Test 1A caused the loss of some target signals. Engineers explained to us that the sensor would eventually be mounted on the lighter, more agile exoatmospheric kill vehicle, which would move back and forth to detect objects that did not initially appear in the sensor’s field of view. The engineers said that Boeing designed software that takes into account the kill vehicle’s normal motion to remove the background noise, but the software’s effectiveness depended on the fast movement of the kill vehicle. Boeing engineers told us that, because of the slow turnaround of the launch vehicle used in the test, the target signals detected during the turnaround were particularly noisy and the software sometimes removed not only the noise but the entire signal as well. The report mentioned that the sensor experienced more false alarms than expected. A false alarm is a detection of a target that is not there. According to the experts we hired, during Integrated Flight Test 1A, the Boeing sensor often mistakenly identified noise produced by the power supply as signals from actual target objects. In a fully automated Appendix I: Disclosure of Flight Test’s Key Results and Limitations discrimination software program, a high false alarm rate could overwhelm the tracking software. Because the post-flight processing tools were not fully developed at the time of the August 13 and August 22, 1997, reports, Boeing did not rely upon a fully automated tracking system when it processed the Integrated Flight Test 1A data. Instead, a Boeing engineer manually tracked the target objects. The contractors realized, and reported to the Ground Based Interceptor Project Management Office, that numerous false alarms could cause problems in future flight tests, and they identified software changes to reduce their occurrence. December 11 Briefing Contractors Report Further on False Alarms On December 11, 1997, Boeing and TRW briefed officials from the Ground Based Interceptor Project Management Office and one of its support contractors on various anomalies observed during Integrated Flight Test 1A. The contractors’ briefing charts explained the effect the anomalies could have on Integrated Flight Test 3, the first planned intercept test for the Boeing exoatmospheric kill vehicle, identified potential causes of the anomalies, and summarized the solutions to mitigate their effect. While some of the anomalies included in the December 11 briefing charts were referred to in the August 13 and August 22 reports, others were being reported in writing for the first time. The anomalies referenced in the briefing charts included the sensor’s high false alarm rate, the silicon detector array’s higher-than-expected temperature, the software’s low confidence factor that it had correctly identified two target objects correctly, the sensor’s lower than expected probability of detection, and the software’s elevation in rank of one target object toward the end of the test. In addition, the charts showed that an in-flight attempt to calibrate the sensor was inconsistent. According to the charts, actions to prevent similar anomalies from occurring or impacting Integrated Flight Test 3 had in most cases already been implemented or were under way. The contractors again recognized that a large number of false alarms occurred during Integrated Flight Test 1A. According to the briefing charts, false alarms occurred during the slow turnarounds of the surrogate launch vehicle. Additionally, the contractors hypothesized that some false alarms resulted from space-ionizing events. By December 11, engineers had identified solutions to reduce the number of false alarms in future tests. Appendix I: Disclosure of Flight Test’s Key Results and Limitations Briefing Charts Include Observations on Higher Detector Array Temperature Some Software Confidence Factors Lower Than Expected Sensor’s Probability of Detection Is Lower Than Expected Software Increases the Rank of One Object Near Test’s End In-Flight Calibration Was Inconsistent As they had in the August 22, 1997, report, the contractors recognized that the silicon detector array did not cool properly during Integrated Flight Test 1A. The contractors reported that higher silicon detector array temperatures could cause noisy signals that would adversely impact the detector array’s ability to estimate the infrared intensity of observed objects. Efforts to eliminate the impact of the higher temperatures, should they occur in future tests, were on-going at the time of the briefing. Contractors observed that the confidence factor produced by the software was small for two target objects. The software equation that makes a determination as to how confident the software should be to identify a target object correctly, did not work properly for the large balloon or multiple-service launch vehicle. Corrections to the equation had been made by the time of the briefing. The charts state that the Integrated Flight Test 1A sensor had a lower than anticipated probability of detection and a high false alarm rate. Because a part of the tracking, fusion, and discrimination software was designed for a sensor with a high probability of detection and a low false alarm rate, the software did not function optimally and needed revision. Changes to prevent this from happening in future flight tests were under way. The briefing charts showed that TRW’s software significantly increased the rank of one target object just before target objects began to leave the sensor’s field of view. Although a later Integrated Flight Test 1A report stated the mock warhead was consistently ranked as the most likely target, the charts show that if in Integrated Flight Test 3 the same object’s rank began to increase, the software could select the object as the intercept target. In the briefing charts, the contractors reported that TRW made a software change in the model that is used to generate reference data. When reference data was generated with the software change, the importance of the mock warhead increased, and it was selected as the target. Tests of the software change were in progress as of December 11. The Boeing sensor measures the infrared emissions of target objects by converting the collected signals into intensity with the help of calibration data obtained from the sensor prior to flight. However, the sensor was not calibrated at the higher temperature range that was experienced during Integrated Flight Test 1A. To remedy the problem, the sensor viewed a star with known infrared emissions. The measurement of the star’s intensity was to have helped fill the gaps in calibration data that was essential to making accurate measurements of the target object signals. Boeing disclosed that the corrections based on the star calibration were Appendix I: Disclosure of Flight Test’s Key Results and Limitations inconsistent and did not improve the match of calculated and measured target signatures. Boeing subsequently told us that the star calibration corrections were effective for one of the wavelength bands, but not for another, and that the inconsistency referred to in the briefing charts was in how these bands behaved at temperatures above the intended operating range. Efforts to find and implement solutions were in progress. April 1, 1998, Report Gap-Filling Software Module Did Not Perform As Designed Assessment Uses Selected Target Signals On April 1, 1998, Boeing submitted a revised addendum to replace an addendum that had accompanied the August 22, 1997, report. This revised addendum was prepared in response to comments and questions submitted by officials from the Ground Based Interceptor Project Management Office, Nichols Research Corporation, and the Defense Criminal Investigative Service concerning the August 22 report. In this addendum, the contractors referred in writing to three problems and limitations that had not been addressed in earlier written test reports or the December 11 briefing. Contractors noted that a gap-filling module, which was designed to replace noisy or missing signals, did not operate as designed. They also disclosed that TRW’s analysis of its discrimination software used target signals collected during a selected portion of the flight timeline and used a portion of the Integrated Flight Test 1A reference data that corresponded to this same timeline. The April 1 addendum reported that a gap-filling module that was designed to replace portions of noisy or missing target signals with expected signal values did not operate as designed. TRW officials told us that the module’s replacement values were too conservative and resulted in a poor match between collected signals and the signals the software expected the target objects to display.