Former F-35 General Manager Explains Key Features Of Developmental Testing

The annual report of the Pentagon’s Director of Operational Test & Evaluation released last month contained a lengthy section on the tri-service F-35 fighter program. One of the issues raised by the report in that section was that the F-35 is lagging in achieving certain “test points” important to defense department assessment of program progress. The notion of test points is not well understood outside the testing community, and I therefore asked Lockheed Martin’s former F-35 Executive Vice President and General Manager Tom Burbage to explain the concept. He responded in an email that was a model of clarity and detail, illuminating not just the concept of test points but the broader testing process. Here is part of what he told me, printed with his permission.

A “test point” is a specifically defined test parameter which can vary by the type of test being performed. Flight test tends to evaluate either Flight Sciences or Mission Systems. As an example, a Flight Sciences test point might be defined by a specific altitude, Mach Number, G, etc. Flight sciences testing generally “clears the flight envelope” for safety reasons (flutter, structural integrity, flight control performance, etc.) or defines the operational envelope of the airplane (max and min speeds, stall performance, turn performance, etc.). Normally, these are measured against the contract specifications. Not all test points are considered equal. Critical test points are those that define the edges of the envelope and are considered very risky for the first pilots attempting to perform the test point. Often, they are accompanied by some incentive compensation for the test pilot. Contractor pilots perform envelope opening test points before government pilots can perform them.

Mission Systems test points evaluate the performance of installed sensors and systems again relative to contract specifications.

There are several other “test points” in a comprehensive test program like F-35. Ground testing includes Static (structural strength) and Dynamic (fatigue life) with production articles in sophisticated ground test fixtures. Most structural deficiencies are found in ground test while most software changes (flight controls and mission systems) are found in flight test as a general rule. Each major sub system also undergoes extensive lab and flight testing governed by specific “test points” related to thermal and vibration tolerance. These feed an assessment of the eventual Reliability and Maintainability of the overall Weapon System. Another unique Test Element of F-35 is the Logistics System which has a separate but very extensive test program with many “test points”.

In a program like F-35, the top level performance of the system is specified in the contract with terms like “tactically significant range” in the case of Mission Systems or with a theoretical aerodynamic performance based on historical tactical fighters in the case of Flight Sciences. These very top level requirements are then devolved through a series of engineering processes into a complex matrix of subsystem performance and flight dynamics. These devolved engineering requirements are then further broken down to specific test points which must be performed to determine that the weapon system meets the overall performance required from an engineering perspective. That is the purpose of Development Testing.

Operational testing then takes the Weapon System into a defined series of operational environments to determine if the system is Operationally Suitable. There has always been a desire to more closely integrate Development and Operational flight testing to try to reduce the time element associated with introducing a new system into operation. There is also a natural rejection of that thought from the OT&E community in their desire to be independent and to receive a fully developed system before beginning their evaluation. In the case of F-35, the founding principles of acquisition reform that were employed by DoD leadership in the early days of the program were all focused on reducing the acquisition timelines. As time progressed and many of those leaders departed the scene, the testing community was successful in changing the fundamental structure of the program. With the program restructuring in the 2010 Nunn McCurdy breach, the F-35 program increased the number of test points and flights by 50%. This extension of test time was a major contributor to the cost growth experienced on the program.