Why Traditional T&E Doesn't Transfer to AI Systems
Developmental test and evaluation exists to characterize system performance against requirements before operational testing. For hardware-dominant systems — an aircraft, a radar, a munition — the test regime is built around repeatability. The same input produces the same output; deviation is a defect; a test matrix captures the performance envelope. Test events are discrete, schedulable, and produce data that maps directly to system requirements.
AI and machine learning components break these assumptions at multiple layers.
Distribution shift is a systemic failure mode, not a defect. A model trained on one operational context will degrade when its deployment environment changes — different lighting, different imagery resolution, different communication latencies, different adversary behavior. That degradation typically does not appear as a hard failure; it appears as a quiet reduction in accuracy that only becomes visible in operational data. Traditional DT&E has no standard method for stress-testing a system against environments it hasn't seen. A test matrix covering the training distribution provides no coverage of the distribution it will actually encounter.
Adversarial inputs create vulnerabilities without visible signatures. Adversarial attacks on machine learning systems can cause systematic misclassification using inputs that are indistinguishable from legitimate ones to a human observer. An imagery classifier can be manipulated to consistently miss a target category through modifications that are invisible at normal resolution. Standard developmental testing does not include adversarial probing as a required test case; that omission leaves a class of failure modes uncharacterized before a system enters service.
Post-deployment learning changes the system the evaluators tested. AI programs that use reinforcement learning or continuous training pipelines may update model weights on operational data, in which case the system that entered OT&E is not the system fielded six months later. Traditional T&E evaluates a fixed configuration. CDAO's OT&E framework acknowledges the underlying issue: AI-enabled capabilities "may require additional testing after fielding due to changes that cause performance drift".
What CDAO's April 2024 Frameworks Established
The Chief Digital and AI Office published four test and evaluation frameworks dated April 2024 (announced May 20, 2024) — covering Model T&E, Human Systems Integration T&E, Systems Integration T&E, and Operational T&E — as a first step toward a community of practice around AI T&E. These frameworks are not binding policy (each states that it does not provide "binding policy and requirements"); they do not amend DoDI 5000.89 or the Services' T&E manuals. What they do is define a structured vocabulary and logic for addressing the failure modes that traditional T&E misses.
The Model T&E framework organizes evaluation around six areas: performance, testing methods, data, AI models, context, and documentation. On robustness specifically, it discusses performance outside the conditions a model was trained on, including natural and adversarial shifts, and notes that adversarial robustness is evaluated through manual or automated red teaming — naming the failure modes that traditional developmental testing does not address. On documentation, it recommends versioning documentation alongside the model and its versioned data, recording data provenance, and maintaining model cards — the baseline continuous-learning programs need before they can trace a performance change to a cause.
The Systems Integration T&E framework addresses a different and underexamined problem: AI components interact with host systems and networks in ways that compound their individual behaviors. A model that performs well in isolation may produce unstable behavior when integrated with a command node that operates under different latency assumptions, or when its outputs feed an automated decision system without a human review step. The Systems Integration framework advises incremental integration testing as the way to catch, troubleshoot, and mitigate undesired emergent behaviors — interactions not produced by any single component in isolation. Program offices that test their AI model in isolation and assume integration adds no new risks have an uncharacterized failure surface at the seam.
The Human Systems Integration T&E framework addresses trust calibration: operators who over-trust AI outputs defer to the system when they should override it; operators who under-trust it create workarounds that negate the system's value. Neither outcome surfaces in a test matrix that evaluates only model accuracy. The Human Systems Integration framework advises that T&E strategies "commit to measuring warfighter trust across operational conditions and evaluating calibration relative to system performance." Applying that advice means measuring whether operators understand what the AI is doing, recognize when it is wrong, and exercise appropriate override judgment, which calls for human-in-the-loop testing with realistic task complexity — a test type with significant cost and schedule implications that TEMPs written before this guidance did not budget for.
The Operational T&E framework addresses performance on representative missions "within an operationally realistic environment against a realistic adversary," calls for metrics that are "operationally relevant, traceable, and understandable," notes that AI-enabled capabilities "may require additional testing after fielding due to changes that cause performance drift," and asks programs to name who is responsible for rollback processes and who has authority to make those decisions.
What MYSTIC DEPOT Reveals About the Remaining Gap
In March 2026, the Defense Innovation Unit, in partnership with the Office of the Director of National Intelligence, put out a solicitation called MYSTIC DEPOT for a vendor-agnostic AI evaluation harness, as DefenseScoop reported and Defense News covered. The solicitation's requirements, as those outlets report them, are a useful statement of what the April 2024 CDAO frameworks left unsolved at the operational level. This is a commercial solutions opening, as DefenseScoop reports, and I found no award announcement, so treat it as a signal of direction rather than an established baseline.
MYSTIC DEPOT calls for the harness to support human-in-the-loop evaluation, agentic behavior testing, and automated red-teaming, including "the execution of adversarial prompts and attack patterns." It requires the harness to be deployable across unclassified, classified cloud, and air-gapped environments, with benchmarks covering unclassified, secret, and top secret workflows — an attempt to let a common evaluation methodology span security domains. It includes requirements for operational stress testing that simulates network degradation in a controlled, reproducible environment. And its benchmark methodology is to address "gaming resistance," which I read as testing whether AI outputs hold up under attempts to systematically manipulate them toward a desired result; that gloss is my interpretation, not the solicitation's wording.
MYSTIC DEPOT's stated aim — a vendor-agnostic harness to standardize testing of AI systems — implies the department does not yet have a common, government-owned evaluation infrastructure; absent one, programs must assemble their own methods and tooling. MYSTIC DEPOT, if delivered, could provide infrastructure for what CDAO's frameworks describe. Until it is available and integrated into the T&E ecosystem, program offices face the practical question of how to apply guidance that assumes tools and methods that aren't yet deployed.
Practical Guidance: Updating Your TEMP Now
The TEMP is the governing document for a program's T&E strategy; it specifies what will be tested, at what fidelity, by whom, and in what sequence. Programs with AI components should treat the following as decisions that need to be reflected in the TEMP before the next milestone review, regardless of whether formal DoDI amendments have been published:
1. Map AI components to the CDAO framework categories. Identify which of the four frameworks apply to each AI component — a supervised classification model in a sensor system has different T&E considerations than a reinforcement learning agent in a logistics optimizer. Document which frameworks apply and why; this creates an audit trail if reviewers ask.
2. Add out-of-distribution and adversarial test cases to the developmental test matrix. Define the operational environments the program anticipates and build test cases that deliberately fall outside the training distribution. Engage with the program's T&E community and the CDAO T&E frameworks on methodology. This does not require MYSTIC DEPOT; it requires test design that explicitly addresses distribution shift.
3. Include human-in-the-loop events in the TEMP schedule and budget. HSI T&E for AI calls for operators, realistic task scenarios, and measurement of over-trust and under-trust indicators. These events take time and are expensive to schedule close to a milestone. Building them into the TEMP early is structurally different from retrofitting them during an OT&E acceleration.
4. Define a configuration management approach for continuous-learning components. If the AI system updates model weights post-deployment, the TEMP needs to specify at what point a model update constitutes a change that requires re-evaluation and what the lightweight re-evaluation standard looks like. The absence of this definition creates ambiguity about what the fielded system is.
5. Build a model card for every AI component in the system. Model lineage, training data sources, version history, and known failure conditions should be documented before integration testing, not reconstructed afterward. The CDAO Model T&E framework recommends versioned documentation and model cards; this is also the minimum foundation for any future adversarial evaluation.
A credible alternative approach is to wait for the DoDI update and MYSTIC DEPOT delivery before revising TEMP structure. The risk is that if milestone reviewers begin asking AI-specific T&E questions before formal policy catches up — a reasonable expectation given CDAO's published frameworks and the department's investment in MYSTIC DEPOT, but a forecast rather than an established practice — programs without a documented approach will be answering them on the reviewer's schedule rather than their own.
The CDAO frameworks also raise a prior-to-integration question every program office should be able to answer: what does a program office actually know about how its AI component behaves outside its training envelope? Programs that can answer that with test data are in a different position than programs that rely on contractor assertions.
Spartan X
The gap between CDAO's guidance and what most program offices can currently execute against it is where technical advisory matters most. Spartan X's Arbiter platform, a multi-model AI verification and governance system, addresses the same question these frameworks pose: how to check what a model produces before anyone relies on it. That makes structured, multi-model verification a practical component of the documentation and evaluation evidence a program office will need as its TEMP takes on AI-specific test events.
Sources and Further Reading
- CDAO AI T&E Frameworks, April 2024 (Internet Archive copies of ai.mil PDFs) — Model T&E, Systems Integration T&E, Human Systems Integration T&E, Operational T&E
- CDAO T&E Strategy and Frameworks overview, May 20, 2024 — ai.mil blog (Internet Archive copy)
- DefenseScoop, March 11, 2026, on the MYSTIC DEPOT solicitation — defensescoop.com
- Defense News, "Pentagon seeks system to ensure AI models work as planned," March 12, 2026 — defensenews.com
- CNAS, "Safe and Effective: Advancing DoD Test and Evaluation for AI and Autonomous Systems," March 2025 — cnas.org
