We Have To Stop Letting The Testers Run The Asylum

Dr. J. Michael Gilmore, head of the Pentagon’s Directorate of Operational Test and Evaluation (DOT&E) reminds me of a radiologist I once consulted. The eminent physician pointed out suspicious dark and light spots on various internal organs. He recommended more scans and even specialized tests to determine the specific nature of each of the many spots that appeared on my x-rays. When I asked my primary physician if I was really that sick, he reassured me. This radiologist had acquired a reputation for finding indications of cancer in the scans of high school athletes undergoing their yearly physicals. As it was explained to me, a radiologist stood no risk of being sued for pointing out something as suspicious which was actually benign. The liability is in failing to identify a spot as worrisome which later turned out to be malignant. Sometimes, good medical practice involves judgment and avoiding running up massive bills chasing low-probability tumors.

This same phenomenon appears to be taking place in DOT&E. This Directorate has an expansive mission. It formulates and issues defense department OT&E policy and procedures; reviews and analyzes the results of OT&E conducted for each major DoD acquisition program; provides independent assessments to the Secretary of Defense, the Under Secretary of Defense for Acquisition, Technology and Logistics and Congress; and oversees DoD acquisition programs to ensure OT&E is adequate to confirm operational effectiveness and suitability of the defense system in combat use. So in essence, DOT&E defines what constitutes appropriate and adequate testing, how much must be done, and how well the defense system must perform before it is considered operationally effective.

Unfortunately, the attitude in DOT&E seems to be that defense systems are built for the purpose of being tested. The Directorate has never met a test point it didn’t like and seems to be of the opinion that if some testing is good, more must be better. Operational testing has become big business, involving tens of thousands of people in program offices, among contractors and DOT&E, costing many millions of dollars and adding months and even years to program timelines.

The Department of Defense has a long history of seeking perfection in defense systems. Too often, the pursuit of the gold standard in system performance results in excessive costs, time slippage and outright failure.

DOT&E appears to have applied this gold standard in its recent report regarding the F-35 Joint Strike Fighter program. In its latest report, DOT&E pronounces the F-35 program as deficient largely because the available software, version 2B, does not permit the use of all the F-35’s systems and potential weapons. It has some impact in the operating envelope for the F-35B. DOT&E warns that this will require those F-35s to get off-board support. The report also warns that there will need to be fixes to the 2B software before it is delivered to the Air Force and that the delivery of the final generation of software, 3F, is likely to be delayed.

How strange, then, that the Marine Corps has declared the F-35B operational with this limited software. Why has the Air Force stood up F-35A training squadrons at several locations? The answer is because as flawed as DOT&E seems to think the F-35 is, the operators believe it to be a substantial improvement over their existing platforms, even when employing early software. Moreover, there is a lot that can be accomplished in the way of pilot training, the development of tactics, operational experimentation and collaboration with other forces that do not require the gold standard of software. The F-35B carries more weapons, farther than the Marine Corps’ aging Harriers. Its stealthiness offers all sorts of opportunities to create new tactics and concepts of operations. Out at Nellis Air Force Base, the tacticians are having their eyes opened to the amazing new things even the software “impaired” F-35 can do. This experimentation is being fed back to the software engineers working on the various versions so that it can be made better still. If you had to wait until the ultimate software version is installed, years of learning, experimentation and even combat effectiveness would be lost.

In reality, all defense systems and particularly military aircraft have been deployed in generations or blocks, each one providing additional capabilities and using enhanced software. Was the F-16A/B a deficient aircraft because it didn’t deploy the same capabilities and sophisticated software as the Block 50? But jumping from one block to the next was easy because the pilots were trained, tactics had been developed and major glitches wrung out of the system prior to the ultimate incarnation being acquired.

Meeting DOT&E test points is not the reason defense systems are developed and deployed. The demands of national security and the evolution of threats have never allowed the U.S. to deploy the gold standard defense system before the fight broke out. The findings in the latest DOT&E report need to be viewed in context. We cannot let the testers run the asylum.