Why AI Incidents Are Different from IT Incidents
Traditional IT incident response is built around availability and integrity failures: a server goes down, a breach occurs, a system produces corrupted data. The failure mode is typically hard — something stops working — and the response is restoration or containment. That model doesn't map well to most AI system failures, which are soft, gradual, and often invisible to standard monitoring.
Silent degradation. A model trained on historical data will drift as underlying patterns change. An unemployment eligibility model trained before a major labor market shift may produce systematically wrong determinations without triggering any alert, because the system is technically operating. Nothing in the monitoring infrastructure flags an accuracy decline until outcome data accumulates — often months later.
Systematic directional errors. A model might be consistently wrong in a specific direction: over-approving fraud claims that match a particular profile, consistently underscoring applicants with non-standard document types, or applying a stale regulatory threshold that has since changed. Identifying the pattern requires outcome analysis, not system status monitoring.
Disparate impact on protected classes. A model that is accurate in aggregate may produce discriminatory outcomes across demographic subgroups. This requires disaggregated outcome monitoring — tracking accuracy by relevant population dimensions — not just overall uptime and error rates. A monitoring setup that tracks only uptime and error rates will miss this.
Data pipeline corruption. If an upstream data source changes its format, encoding, or coverage, model inputs can become meaningfully different from training data without any visible error. The model keeps running; its outputs silently degrade. This is a risk in benefits systems that consume federal data feeds.
Effective gaming by bad actors. In fraud detection contexts, adversaries who observe decision patterns can adapt inputs to exploit decision boundaries. This looks like a system doing its job while the underlying effectiveness erodes.
Each of these failure modes requires different detection methods, different evidence standards for causation, different remediation strategies, and different communication obligations to citizens and oversight bodies. They don't fit into a standard major incident runbook.
The MANAGE Function and What It Actually Requires
The NIST AI Risk Management Framework (NIST AI 100-1, January 2023) organizes AI risk management into four functions: GOVERN, MAP, MEASURE, and MANAGE. Where state governance frameworks adopt the AI RMF — and the framework is voluntary, though New York's NYS-P24-001 references it — in our assessment, the emphasis tends to fall heavily in GOVERN (policies, roles, accountability structures) and MAP (system inventories and risk categorization). MEASURE gets partial attention through pre-deployment testing and procurement assessments. (NIST states that AI RMF 1.0 is being revised under the White House AI Action Plan; 1.0 remains the current published version.)
MANAGE is where ongoing operational responsibility lives. It covers risk prioritization and response based on monitoring results, implementation of strategies to maximize benefits and minimize negative impacts throughout the system lifecycle, management of third-party AI risk, and documented recovery and treatment protocols. These aren't aspirational — they're operational capabilities that require staff, procedures, tooling, and clear decision authority.
A state that issues an AI executive order, builds a use-case inventory, and requires vendor compliance certifications has addressed GOVERN and parts of MAP. It has not addressed MANAGE unless it has people who can detect a model drift event, authority structures that allow them to act, and documented procedures for what happens when they do.
The enforcement gap in state AI policy reinforces this. A September 2026 Smarsh analysis found that 49 of 51 jurisdictions (the 50 states plus D.C.) have issued agency AI guidance, but fewer than half have active enforcement mechanisms such as registries, audits, or reporting channels (many delegate enforcement to agency leadership), and only nine spell out explicit consequences for non-compliance, though Smarsh notes that most states without stated penalties still enforce through approval gates, access revocation, and standard HR discipline. In our assessment, these gaps often reflect the same underlying problem: policy that describes expected outcomes without operational mechanisms for monitoring them.
Classifying State AI Incidents
The operational gap starts with definition. Neither Arkansas's nor New York's policy defines an "AI incident", and without a definition there is no trigger for response and no clear ownership for investigation. Spartan X recommends a classification organized by consequence, not cause. The timelines below are proposed starting points, not regulatory requirements:
Tier 1 — Immediate response. Failures affecting safety, liberty, or access to essential services: a benefits system denying claims in a pattern that suggests disparate impact on a protected class, a child welfare tool producing systematically incorrect removal recommendations, a fraud detection engine misclassifying a protected characteristic as a risk proxy. Response is immediate human review override, system pause if the failure cannot be isolated to a specific input category, and formal incident documentation within 24 hours. These systems include anything that informs or produces decisions about eligibility for essential programs, child placements, or criminal justice proceedings.
Tier 2 — Time-bounded investigation. Failures affecting service accuracy or program integrity where citizens are harmed but the failure is not acute: an eligibility model whose approval rate has shifted materially without a corresponding policy change, a procurement scoring system producing unexplained variance from prior benchmarks, a workforce screening tool that produces anomalous outcomes for a specific document type. Investigation within five business days, with interim human review for decisions in the affected category during the investigation period.
Tier 3 — Documented drift. Gradual accuracy degradation within tolerable thresholds but outside the monitoring bounds established at deployment. Requires root cause analysis and a remediation plan within 30 days, vendor notification if the model is vendor-operated, and updated performance baselines. No immediate operational pause required, but documented and tracked.
Every AI system in a state's production inventory should have a tier assignment based on its use case, documented monitoring baselines established at deployment, and a clearly named incident owner. Without an owner, incidents get discovered during post-hoc audits rather than active monitoring — and the delay between failure and detection is where citizen harm can accumulate.
The Vendor Dependency Problem
The practical complication is that where an AI system is vendor-operated rather than state-controlled, it creates response problems that governance frameworks can miss:
Visibility gaps. The state may have no direct access to model performance metrics, training data versions, or inference logs unless the vendor contractually provides them. Contracts that describe accuracy at procurement but don't require ongoing monitoring transparency leave the state operationally blind to model changes.
Rollback authority. If a vendor pushes an unannounced model update — which SaaS-delivered AI products can do — the state's ability to roll back to a prior version depends entirely on whether the contract specifies that right. Contracts that do not specify rollback rights leave the state without one.
Response timeline mismatches. A state's incident response procedure may require a 24-hour Tier 1 response. A vendor SLA may promise different terms. The gap becomes a governance failure in an actual incident, especially when the vendor is the only party with access to root cause information.
Root cause access. Investigating a disparate impact failure requires access to the model's decision logic at the input level. Black-box models operated under commercial licenses may contractually bar that access — and even when they don't, vendors have limited incentive to facilitate it quickly.
These gaps are straightforward to address at the contracting stage: require monitoring log access, define rollback rights explicitly, establish incident notification timelines, and specify what root cause investigation access the state retains. They are significantly harder to negotiate retroactively, especially under an active incident.
Two 2026 policy actions show how far some states have already gone. Arkansas's Executive Branch AI Acceptable Use Policy (effective February 26, 2026) applies to "all Executive Branch employees, contractors, vendors, and third parties who design, procure, implement, or use AI tools or systems," and requires that AI be "designed, tested, and monitored to prevent bias or disparate impacts." New York's NYS-P24-001 (issued 2024, updated June 24, 2026) extends to vendors and contractors, makes system owners responsible for "periodically assessing the outputs of their in-production AI systems," and requires annual AI risk assessments. Neither policy defines an AI incident, sets response timelines, or assigns incident ownership (New York does require a named Information Owner for each AI system, but gives that role no incident-response duties). The next operational step is turning monitoring obligations into an incident response procedure: triggers, tiers, owners, and vendor notification terms.
Building the Capability Before the Incident
Constructing an AI incident response capability is a three-part operational problem.
Monitoring baselines for every production system. At deployment, document the system's expected performance: overall accuracy by output category, performance disaggregated by relevant population dimensions, approval and denial rate stability within defined bounds, and data pipeline health indicators. The baseline is the threshold that triggers escalation. Without one, drift has no reference point and Tier 2 incidents are invisible until they become Tier 1.
An ownership map. For each Tier 1 and Tier 2 system, identify in writing: who calls the incident (incident owner), who investigates the technical cause (data or IT team), who holds remediation decision authority (agency IT director or CISO), and who handles external communication to affected citizens and oversight bodies. This map needs to exist before an incident, not during one. When authority is undefined, each party can assume someone else owns the response.
A tabletop exercise. Use a specific production system. Introduce a plausible failure scenario — demographic accuracy divergence detected in quarterly outcome review, or an upstream data feed change that went unnoticed for 60 days. Walk the team through detection, tier classification, investigation initiation, vendor engagement, and communication obligations under actual response timeline pressure. Tabletop exercises surface authority gaps, vendor notification delays, and unclear tier boundaries that policy documentation obscures. They also establish that the monitoring infrastructure can actually detect the failure mode in question — an assumption that is easy to get wrong until it is tested.
The likely result for a state that skips this work: the policy exists, the inventory exists, but nobody owns the pager when it rings.
Sources and further reading
- NIST AI Risk Management Framework (AI 100-1), National Institute of Standards and Technology, January 26, 2023: https://doi.org/10.6028/NIST.AI.100-1
- NIST AI RMF Playbook (Manage function action examples), NIST Trustworthy and Responsible AI Resource Center: https://airc.nist.gov/airmf-resources/playbook
- Arkansas Executive Branch AI Acceptable Use Policy, effective February 26, 2026: https://sas.arkansas.gov/wp-content/uploads/Policy_AI-Acceptable-Use.pdf
- New York State AI Acceptable Use Policy NYS-P24-001, issued January 8, 2024, updated June 24, 2026: https://its.ny.gov/acceptable-use-artificial-intelligence-technologies
- Smarsh, "How U.S. States Regulate Government Use of AI" (analysis of state AI enforcement): https://www.smarsh.com/blog/thought-leadership/state-ai-regulation-government-use
- CISA, "Guidelines for Secure AI System Development" (joint guidance, November 2023): https://www.cisa.gov/news-events/alerts/2023/11/26/cisa-and-uk-ncsc-unveil-joint-guidelines-secure-ai-system-development
Spartan X's Arbiter work in multi-model AI verification addresses part of this gap: independent checks on AI outputs and an auditable record that helps agencies reconstruct what a system decided and why. For state agencies running AI in high-consequence programs, the same verification discipline applies to the monitoring baselines, incident classification, and vendor accountability structures described here.
