The documentation problem is a governance problem
AI is embedded in state program administration at a depth most state CIOs cannot fully account for. Eligibility platforms flag applications for manual review based on model outputs that arrived as a vendor feature update, not a deliberate procurement decision. Fraud detection tools score claims before a human examiner sees them. Document processing pipelines classify intake submissions and route them to caseworkers. Chatbot triage handles the first tier of call center traffic.
Most of these deployments happened through agency-level procurement, often as add-on capabilities within enterprise vendor contracts. They were evaluated as product features, not AI systems. The result is that many state agencies are now running consequential, model-driven logic against program populations without being able to answer the basic governance questions: what systems are running, what decisions they influence, how they were validated, and who is accountable when one produces a wrong output.
This is not a technology problem. The systems are running. It is a documentation and governance problem — and that gap is becoming an audit exposure.
What federal oversight frameworks are beginning to expect
OMB Memorandum M-25-21, issued April 3, 2025, requires federal agencies to retain or designate a Chief AI Officer, requires CFO Act agencies to convene an AI governance board, requires each agency to inventory its AI use cases at least annually, submit the inventory to OMB, and post a public version, and sets minimum practices — including pre-deployment testing, ongoing monitoring, and human oversight — for high-impact AI. Federal agencies are not state agencies: the memo states that it governs only agencies' own use of AI and does not create rights or obligations for the public, so it does not directly bind a state Department of Health or a state labor agency.
But the federal agencies that oversee major joint federal-state programs — the Centers for Medicare and Medicaid Services for Medicaid and CHIP, the Food and Nutrition Service for SNAP, the Department of Labor for unemployment insurance — are covered by M-25-21 and must apply it to their own systems. When those agencies conduct program reviews of state-administered programs, the documentation practices they now apply to their own systems are likely to shape the questions they ask about yours. That is our expectation, not a requirement the memo imposes on states.
The compliance mechanism is not primarily regulatory. It is practical and operational:
- Single audit requirements. Non-Federal entities, including states, that expend $1,000,000 or more in federal awards in a fiscal year must have a single or program-specific audit under 2 CFR §200.501. The Federal Register notice (89 FR 30046, April 22, 2024) raised the threshold from $750,000 to $1,000,000, effective October 1, 2024; the FAC submission guide applies the $1,000,000 figure to fiscal years starting on or after that date. Under 2 CFR §200.514(c), the auditor must perform procedures to understand internal control over Federal programs sufficient to plan the audit to support a low assessed level of control risk of noncompliance for major programs. Whether an auditor treats AI components embedded in program systems as part of that review is our inference, not something the regulation says, but it is a reasonable question for auditors to raise.
- Program review walkthroughs. Federal program reviews can include systems walkthroughs for eligibility and case management platforms. A reviewer asking what automated tools affect eligibility determination is asking about AI. If the agency cannot answer specifically, the reviewer notes an information gap — one that generates follow-up requests and can escalate into a formal finding.
- Conditions of participation. Federal programs include documentation, security, and systems requirements as conditions of state participation. CMS's Acceptable Risk Controls for ACA, Medicaid, and Partner Entities (ARC-AMPE), which supersedes and replaces the earlier MARS-E standards, for example, sets security and privacy control standards for ACA Administering Entities — defined to include state and federal Exchanges, state Medicaid agencies, state CHIP agencies, and state agencies administering a Basic Health Program. As AI components become embedded in those systems, documentation expectations are likely to follow.
The state that cannot answer a reviewer's question about its AI systems does not necessarily fail an audit on that basis alone. But it generates findings, corrective action requests, and additional scrutiny for every claim it cannot support — and those findings carry into the next review cycle.
What the documentation gap actually looks like
The documentation gap is rarely that nothing exists. It is that what exists is fragmented, vendor-held, or scoped to the wrong question.
A typical state agency holds:
- Vendor contracts — but not documentation of what model versions run, against what training data, or when the vendor last updated the underlying model
- Change request and help desk logs — but not documentation of how system decision logic changed when the vendor pushed a model update
- Testing and acceptance records — from initial deployment, not from subsequent vendor updates
- Security review documentation — scoped to the system boundary and data flows, not to the AI component specifically
What a federal program reviewer or auditor needs differs in kind, not just in depth:
- What AI components are embedded in which program functions
- What the system produces — a score, a flag, a classification, a recommendation — and how that output connects to a human decision
- What population and time period the system was validated against, and what error rate targets were established
- What safeguards exist for high-risk outputs: human review thresholds, appeal pathways, override documentation
- Who at the agency owns accountability for each system and how changes are authorized
The gap between what a state typically holds and what a reviewer needs is what an AI use-case inventory closes.
What a useful AI inventory actually contains
The NIST AI Risk Management Framework, published January 26, 2023, defines four core functions — Govern, Map, Measure, Manage — that structure how organizations document and manage AI across its lifecycle. The Map function establishes the context for each system — its purpose, setting, and where risk concentrates — and NIST states that Map outcomes are the basis for Measure and Manage. The inventory itself appears under Govern: subcategory GOVERN 1.6 calls for "mechanisms ... to inventory AI systems." A use-case inventory is where those two functions meet — the governance mechanism that holds each system's Map context. NIST says AI RMF 1.0 is being revised as part of the White House AI Action Plan, so check NIST's AI RMF page for the current version.
For a state program office, a practical inventory entry contains:
System identification. System name, vendor, version, and deployment date. The program or function it operates within. A brief description of what the system does.
Decision scope. What outputs the system produces — score, classification, flag, recommendation. What human or automated decisions those outputs connect to. Which program population is affected and at what volume.
Data handling. What personally identifiable information categories the system processes. Where input data originates and where outputs flow. Data retention and deletion practices relevant to the AI component.
Validation basis. What population and time period the system was validated against. What performance targets — accuracy, false positive rate, false negative rate — were established at deployment. When the system was last assessed for performance drift against current program populations and conditions.
Accountability. Named technical owner inside the agency with authority over the system's configuration and operation. Named program owner with authority over whether and how the system is used. Escalation path for disputes about AI outputs.
Human oversight. What thresholds or conditions trigger human review of AI outputs. What the human review process requires of the reviewer. How overrides are documented.
Each entry should be completable by the agency from its own records — not from the vendor's assurances. Entries where fields cannot be filled from agency records identify specific documentation gaps that are operational risks independent of any audit.
Five scenarios to design the inventory against
Federal program review. A CMS, FNS, or DOL team reviews program administration and asks what automated tools are used in eligibility determination, fraud detection, or case management. In our estimate, the question is answerable in under an hour from a current inventory — or it triggers a multi-week scramble across vendor contracts and program staff.
Constituent complaint or advocacy inquiry. An applicant claims their benefits were incorrectly denied or delayed by an automated system. The agency needs to reconstruct what the system produced, when, and why — and demonstrate that a human review pathway existed and was available. The inventory identifies where to look and what the oversight process requires.
Vendor contract renewal. The agency evaluates whether an AI component of an enterprise system is performing as required and whether vendor model updates have materially changed the system's behavior. Without a validation baseline in the inventory, the agency has no independent basis for assessing the vendor's characterization of its own system.
Single audit internal controls finding. An auditor identifies a gap in how AI systems are governed in federally-funded program administration. The corrective action plan must name the affected systems, the specific gaps, and remediation steps with timelines. The inventory is the starting point for any coherent corrective action.
Legislative inquiry. A state legislative committee asks what AI systems the agency uses and what decisions they affect. An accurate, organized response — possible only from existing documentation — signals institutional competence. An inability to answer specifically signals something else.
Building the inventory: practical sequence
In our estimate, for most agencies a functional AI inventory is a three-to-six month effort at deliberate pace — longer if AI is embedded in vendor systems without contract provisions giving the agency access to technical documentation about those components.
1. Define scope. Establish what the agency means by "AI" for inventory purposes. A working definition for program administration contexts: any vendor or agency-built system that uses machine learning, statistical modeling, or rules derived from training data to produce outputs that influence program decisions or case routing. This definition includes deployed features inside enterprise platforms, not only standalone AI tools procured as such.
2. Issue a structured survey to each program area. Ask what AI-capable tools are in use, who holds the vendor contract, what program functions the tool affects, and who the technical and program contacts are. Budget two survey rounds — the first will undercount, because program staff do not routinely identify vendor product features as AI systems.
3. Review contracts for documentation provisions. Identify which contracts require the vendor to provide information about AI components — model update notifications, training data documentation, performance reporting, or audit access rights. Contracts without these provisions represent documentation gaps the agency cannot close without renegotiation. Flag them for the next renewal cycle.
4. Apply a risk tier to each system. Systems that affect individual eligibility determinations, fraud findings, or benefit amounts require more documentation depth than systems used for operational analytics or aggregate reporting. A three-tier classification — high (individual program determinations), medium (case routing and prioritization), low (internal operational) — focuses the inventory work on where accountability exposure is greatest.
5. Assign documented accountability. Name the technical and program owner for each system in writing, with a defined escalation path. Accountability that exists only informally does not satisfy a corrective action request and provides no operational clarity when a system behaves unexpectedly.
6. Establish a review and update cycle. The most common inventory failure is not the initial build — it is the first update cycle that does not happen. Require program areas to report material changes to AI system configurations or vendor model updates, and schedule an annual inventory review as a standing governance activity.
The decision the inventory enables
An AI inventory is not itself a safeguard. It does not validate that deployed systems are performing correctly or that human review processes are adequate. What it does is establish the documentation baseline without which no meaningful governance is possible.
A state that can name its AI systems, describe what they do, identify who owns them, and explain what oversight exists is in a fundamentally different position than one that cannot — with federal reviewers, with program populations who interact with those systems, and with state leadership making decisions about AI investment and risk.
The honest question a state CIO needs to answer is not "should we document our AI systems" but "what would we need to hand a federal reviewer today, and can we produce it?" Running that question against each major program area without prior preparation will identify the documentation work that is already overdue.
Spartan X's AI assurance and verification work centers on the same questions an inventory forces: what a model-driven system produces, how its outputs reach a human decision, and what evidence shows it is performing as claimed. For a state program office, the inventory is where that work has to begin.
Sources and further reading
- Office of Management and Budget, M-25-21: Accelerating Federal Use of AI through Innovation, Governance, and Public Trust (April 3, 2025) — current operative federal AI governance memo; requires CAIO designation, AI use-case tracking and reporting, and minimum practices for high-impact AI systems.
- NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (January 26, 2023) — the Govern, Map, Measure, Manage framework that structures organizational AI documentation.
- Office of Management and Budget, Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards (2 CFR Part 200) — single audit requirements; §200.501 threshold raised to $1,000,000 effective October 1, 2024 (89 FR 30046, April 22, 2024).
- CMS, ARC-AMPE Volume I: Acceptable Risk Controls for Affordable Care Act (ACA), Medicaid, and Partner Entities, v1.02 (v1.0 March 4, 2025; v1.02 April 10, 2025) — supersedes MARS-E; security and privacy control standards for ACA Administering Entities, including state Medicaid and CHIP agencies.
