Statistical significance

The Pentagon's top weapons tester singled out three major programs -- the F-35, the Littoral Combat Ship and the Joint Light Tactical Vehicle -- in a keynote address last month on the importance of utilizing statistical analysis to support operational testing.

Director of Operational Test and Evaluation J. Micheal Gilmore presented this briefing -- on April 11, 2016 in Arlington, VA -- to a workshop organized by the Defense Department and NASA for professional testers from the military, NASA, national laboratories and across the federal government.

Army Maj. Roger Cabiness II, Gilmore's spokesman, provided additional background:

The briefing was the opening keynote for the Rigorous Test and Evaluation Workshop, which the DOD test and evaluation community planned collaboratively with the Statistical Engineering Group at NASA. The primary objective of the workshop was to provide a forum for the professional community to share and discuss statistical approaches to test design and evaluation, including the characterization of system capabilities. The workshop was a continuing part of the DOT&E and NASA Statistical Engineering Agreement, which we use to share best practices across organizations.

DOT&E has made it a top priority to ensure that operational tests are adequate, not only in terms of how much testing is conducted, but also under what conditions. Statistical thinking and the associated methods are an essential element in determining of test adequacy. DOT&E has advocated for the use of statistical methodologies, including Design of Experiments (DOE), reliability test planning, survey design, and rigorous statistical analyses. These methodologies not only provide a rigorous and defensible coverage of the operational space; they also allow us to quantify the trade-space between the amount of testing and the precision needed to answer the complex questions about system performance. They allow us to know, before conducting the test, which analyses we will be able to conduct with the data and therefore, what questions about system performance we will be able to answer. Finally, they provide the analytical tools to answer the question of how much testing is enough in the context of uncertainty.