What a demonstration leaves unanswered
A software RFP can evaluate functional requirements, cost, delivery schedule, vendor stability and security posture. Keep those checks. Add evidence about the model, its data and its intended operating conditions. AI systems can produce repeatable outputs: a fitted decision tree, for example, applies learned decision rules to an input. Conventional software can also behave differently when dependencies, execution order or inputs change. Repeatability alone does not establish that either system is correct or that a demonstration covers the agency's workload.
Three evaluation questions deserve particular attention. The NIST AI RMF Playbook's Measure function recommends context-sensitive evaluation, assessment of whether results generalize, and monitoring of performance over time:
Distribution shift. A model may perform well in testing and less reliably when the operating environment differs: applicants submit different documents, a program rule changes, or a new region uses different record formats. A vendor may already have tested these variations; the procurement team should examine that evidence instead of assuming it exists or is absent. Ask how training, validation and final evaluation datasets were separated, which conditions were tested, and how those conditions compare with the state's current caseload. Measure 1.2 specifically calls for examining whether measurements generalize beyond the context in which they were obtained.
Performance gaps hidden by averages. An overall accuracy score can hide concentrated errors in a subset of cases. As a hypothetical example, a document-processing model might handle standard payroll records reliably but make more errors extracting income from self-employment records. That is a documentation difference, not a demographic characteristic. Testing supported languages, document types and operating conditions separately can reveal where additional validation or human review is needed. Demographic breakdowns may also help evaluate performance for the people an agency serves, where relevant to the use case and supported by appropriate data. A measured difference does not, by itself, establish its cause or justify different eligibility standards. The About discussion in the NIST AI RMF Playbook, Measure 1.3 explains why evaluations should look beyond averages for concentrated failures.
Performance changes over time. A deployed model can remain unchanged while its inputs or operating environment change. As hypothetical examples, a fraud-detection model could encounter different claims patterns after a disruption, or a document-processing model could encounter new fields after a rule change. The procurement question is whether the contract requires useful monitoring, reporting and corrective action when performance falls outside agreed limits. Check the actual terms; do not assume that monitoring is included or absent. Measure 2.4 addresses monitoring deployed AI systems in their production context.
What NIST's AI Risk Management Framework gives procurement teams
The NIST AI Risk Management Framework 1.0 (January 2023) organizes AI risk management around four functions: Govern, Map, Measure and Manage. NIST describes it as voluntary, non-sector-specific and use-case-agnostic. It supplies a structure for managing risk, rather than a legal certification or a mandatory checklist for every purchase. A state procurement team can use that structure to develop evaluation criteria appropriate to its program.
The RMF's most useful contribution to procurement is its treatment of the full deployment context: who the AI system affects, what happens when it produces an error, and how harm can be detected and corrected. That framing translates directly into RFP questions that vendors either can or cannot answer.
Four areas to tailor to the proposed system's role, consequences and procurement scope:
1. Validation documentation with useful breakdowns. Request the vendor's evaluation methods, results and limitations. Choose metrics that fit the task: precision and recall may help evaluate a classifier, while a document-extraction tool needs measures of field-level correctness and omissions. Specify relevant breakdowns by document type, supported language, operating condition and, where justified, demographic characteristics. Record sample sizes and uncertainty so a small subgroup is not mistaken for reliable evidence. Missing evidence creates a question to resolve through testing, additional documentation or a narrower deployment scope; it does not establish that every product requires the same demographic report. Measure 1.1 calls for methods appropriate to the risks being assessed.
2. Data lineage and scope. Request documentation of training and evaluation data sources, covered periods, relevant populations and known gaps, subject to applicable confidentiality and privacy constraints. For benefits eligibility and program-integrity tools, ask whether the evaluation represents current rules and case documentation. A different training period warrants investigation; it does not prove a particular error pattern. If a supplier cannot disclose underlying data, assess what independent testing, controlled access or other evidence can substantiate its performance claims, and record the remaining uncertainty.
3. Monitoring and drift detection plan. Require a written post-deployment plan covering how the vendor will detect model drift, what performance metrics will be reported to the state and on what cadence, and what thresholds trigger a mandatory revalidation event. Specify response and remediation deadlines, responsibility for investigation, and remedies during contract performance as well as at renewal. These are recommended contract terms to tailor with procurement and counsel. Put the agreed duties and remedies in the contract so the agency can verify and enforce them.
4. Review, correction and appeals integration. For a system that influences benefits determinations, map its role to the program's actual notice, review, appeal and recordkeeping requirements. Specify who can correct an error, what authority they have, and what evidence they can inspect. Ask the vendor to demonstrate how its output enters the case record and supports agency review. These are procurement recommendations that must be tailored to the workflow; NIST does not impose a universal legal right to human override for every AI-assisted action.
The explainability requirement
Identify the notice and appeal rules for the specific program before writing an explanation requirement. For example, 42 CFR 431.210 specifies content for Medicaid notices, including the intended action and its effective date, reasons, supporting regulations or a change in law, hearing rights and circumstances in which benefits continue pending a hearing. That program-specific rule should not be presented as a uniform model-explanation requirement across all state services.
For a benefits workflow, require a demonstration showing how the agency produces an accurate, understandable account of its decision and supports any required notice. A model-generated explanation alone does not establish compliance. The agency must connect the relevant case evidence, applicable program rules and decision process, with counsel confirming the requirements for that program.
"The algorithm determined..." does not tell a caseworker why an outcome is correct. A useful acceptance exercise should show: (a) which case facts and program rules support the decision, (b) how a reviewer can identify and correct an erroneous input or output, and (c) how the result integrates into the required notice. Internal model weights may not provide that explanation, and the cited Medicaid provision does not require their disclosure. Evaluate whether the complete workflow supports the agency's obligations before approving its intended use.
Pre-award technical evaluation checklist
Before shortlisting an AI vendor, a state procurement team should identify how these questions will be answered through vendor documentation, agency testing or independent evidence, with consequential gaps resolved before acceptance:
- Does the evaluation use task-appropriate metrics and relevant breakdowns, with sample sizes and limitations, to test the state's actual workloads?
- What is the training data time range and source, and how comparable is it to the state's current caseload?
- What is the vendor's post-deployment monitoring process, and what performance reporting does the state contractually receive?
- How does the system generate an explanation for an adverse determination, and what does that explanation include?
- What contractual provisions govern model updates — specifically, what triggers a notification requirement, and does the state retain the right to retest before a major model revision goes live?
- Who owns the model output records, and what data retention obligations does the vendor assume?
Use the answers to identify what is demonstrated, what needs an acceptance test, and what remains uncertain. An unanswered question can reflect a gap in the product, its documentation or the requested scope. Resolve consequential gaps before accepting the corresponding operational risk.
Put the evidence through an acceptance exercise
Consider a hypothetical purchase: an agency wants AI to extract income information from documents for a caseworker to review. The intended role is extraction assistance; authority to determine eligibility stays where the program requires it. A polished demonstration of a clean payroll statement leaves the harder procurement question open: what happens when the system encounters an incomplete record and the employee is working through a queue?
The following is a recommended test sequence, not a NIST-mandated procedure:
- Define a bounded task and a reference answer. Specify the fields the system should extract and what it should do when evidence is missing or contradictory. Have qualified program staff establish the expected results from the source documents and resolve disagreements. Otherwise, the test can measure agreement with an unreliable answer key.
- Build a representative, protected test set. Include the document types and difficult conditions the agency expects to encounter, with an explicit list of conditions the sample does not cover. Use data the agency is authorized to use, with appropriate handling controls. Keep final acceptance cases separate from examples used to tune the system. If the supplier adjusts the product after seeing failures, use fresh held-out cases to assess the correction.
- Examine errors and their consequences. A correctly extracted name does not offset an incorrect income amount simply because both count as fields. Review which errors could change a decision, which omissions create extra work, and whether the system identifies uncertainty or presents a wrong answer confidently. Choose thresholds before scoring the final test, based on the task and its consequences.
- Test the employee's recovery path. Give a reviewer a deliberately incorrect extraction. Can the reviewer find the source, correct the field, see downstream effects and preserve the correction in the case record? Observe whether the interface makes careful review practical. A human approval button alone does not show that the employee has the information or time to catch an error.
- Record the decision and its limits. Accept the defined scope, require remediation and retesting, restrict the deployment, or decline it. Name the agency official who accepts residual risk and record why the evidence supports that decision. Approval for one document-processing task should not silently become approval for an automated determination.
There is a legitimate narrower option when evidence is incomplete: a limited pilot using appropriately controlled data and outputs that do not drive consequential decisions. Its purpose should be to answer named uncertainties, with an end date and exit criteria. It should not become an indefinite route around acceptance requirements.
Read a performance number as a staffing decision
For a classifier, precision and recall answer different operational questions. Precision describes how many positive predictions are correct; recall describes how many actual positive cases the system finds. Their formal definitions help explain why one score cannot settle a procurement decision.
In a hypothetical fraud-screening workflow, a flag should be evaluated as an investigative lead, not proof of fraud. A configuration that finds more potentially relevant cases may also send more incorrect flags to investigators. Ask for the underlying counts, the decision threshold and the review workload at that setting. Then ask what happens to cases the system misses. The agency needs to choose a workable balance for its mission; there is no universal score that makes that choice for it.
For document extraction, a different measurement may be more useful: consequential field errors, omissions and correction time. Avoid purchasing a metric that looks impressive while leaving the actual service constraint unmeasured. Review capacity, source access and correction authority belong in the same acceptance discussion as model quality.
Red flags in vendor responses
Three response patterns that should pause evaluation before a vendor is shortlisted:
Proprietary system with no usable evaluation evidence. A supplier can protect intellectual property while providing meaningful evidence of performance. Seek validation methods and results, controlled testing or independent assessment appropriate to the purchase. If no usable evidence is available, the agency lacks a defensible basis for accepting the claimed model quality. Document the gap and its consequences instead of treating a proprietary architecture itself as proof of poor performance.
Aggregate accuracy without workload detail. An overall number is insufficient when it leaves a material risk in the intended workflow untested. Ask for the relevant breakdowns and enough information to interpret them. For a document-processing tool, that might include document formats and missing-field errors; for a resident-facing system, supported languages and relevant population characteristics may also matter. The required evidence follows the use case, rather than a single checklist imposed on all government AI.
Updates without agreed change controls. If model changes can affect consequential outputs, seek terms defining notification, impact assessment, testing, rollback and agency acceptance where warranted. Routine fixes and material behavioral changes may need different treatment. Establish that distinction before award so the agency and supplier know when additional validation is required.
The RFP language gap
An RFP is an opportunity to make evaluation expectations explicit before proposals arrive. A supplier may provide useful evidence without being asked, and requirements may also appear in other contract documents. The agency should nevertheless check that the complete procurement establishes the evidence, monitoring reports and remedies it needs. Leaving those matters unresolved creates avoidable uncertainty at acceptance and during operation.
The NIST AI RMF Playbook — the companion implementation guide — translates the RMF's functions into specific actions and recommended activities. State procurement teams don't need to develop AI evaluation criteria from first principles. They need to translate existing federal frameworks into the RFP language their contracting offices will act on.
One practical option is to designate a qualified evaluator to examine AI performance evidence separately from the functional demonstration, consistent with the agency's procurement rules and available expertise. Measure 1.3 recommends involving internal experts who were not the system's front-line developers or independent assessors in evaluation. Give that reviewer a defined scope, access to evidence and responsibility for documenting unresolved risks. This strengthens the technical basis for the decision without assuming every agency needs the same committee structure.
Sources and further reading
- NIST, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, January 2023: doi.org/10.6028/NIST.AI.100-1
- NIST AI RMF Playbook, AI Risk Management Framework companion: NIST AI RMF Playbook
- Medicaid notice content requirements, 42 CFR 431.210: ecfr.gov/current/title-42/section-431.210
Spartan X's AI advisory, evaluation and governance capabilities can help an agency connect its mission needs with the technical questions a vendor should answer. That expertise can support a clearer evaluation before award or a focused assessment of an existing system.
