Evaluating an Automation Pilot: Acceptance Tests and Failure Recovery

Define completion in observable terms

An automation pilot should be accepted against the work it must complete, the boundaries it must respect, and the failures it must handle. Write cases in terms of an input, a required outcome, and evidence of the result. A successful model response is only one component if the workflow also retrieves information, updates a record, or hands work to staff.

An illustrative enquiry workflow might require a valid message to become one case in the correct queue with the approved summary fields. A separate case might require an incomplete message to remain unsubmitted until a missing detail is resolved. These outcomes are observable. A statement that the agent understood the request does not establish either result.

Cover normal, ambiguous, and unauthorised inputs

Include expected inputs, missing fields, contradictory information, invalid destinations, unsupported requests, and repeated submissions. Test user roles and resource boundaries separately from language quality. A user asking confidently for a restricted action should not receive more authority than their account permits. Unknown operations should stop or route to a defined review process.

Also include instructions hidden in external documents, tickets, or retrieved pages. OWASP explains why untrusted content needs controls beyond a conversational instruction. Test whether the application enforces permissions even when generated output proposes an inappropriate action. The expected evidence is a blocked or constrained operation with a clear user or operator outcome, rather than a reassuring sentence in the conversation.

Distinguish failure states before retrying

A connector timeout may mean an action never began, completed without a response, or remains unresolved. Those states require different recovery. Track a task identifier and the known execution state so an operator can investigate. Where possible, use an idempotency mechanism: repeating the same approved operation should not create an extra effect.

An illustrative support connector submits a ticket but loses the response. Blindly repeating the submission may create a duplicate. A safer design checks for the existing operation using a stable identifier or sends the item to reconciliation when the outcome cannot be established. The acceptance test should verify that uncertainty is represented honestly and that the student is not shown an invented case number.

Test recovery as part of the pilot

Exercise expired access, unavailable providers, invalid outputs, interrupted runs, and partial completion. Confirm that the system stops within agreed limits and records enough information to identify affected items. An error message should say what happened and what the next actor can do. Avoid storing unnecessary personal details just to create extensive diagnostic logs.

Decide whether recovery means retrying, resuming from a checkpoint, reversing a reversible update, or asking staff to reconcile the record. Some external actions cannot simply be undone, so the process must define correction rather than promise rollback. Have the intended operator perform the recovery using the handover instructions. This tests whether the procedure is usable beyond the development team.

Compare total work and output quality

Evaluate the pilot against a representative baseline of the existing process where feasible. Include preparation, staff review, exceptions, corrections, and downstream rework. Record which cases were excluded and why. An automation that completes routine cases quickly may still fail the institution's acceptance standard if its errors create disproportionate follow-up work.

Keep serious boundary failures separate from aggregate performance. The pilot report should show the tested sample, observed outcomes, unresolved issues, and limitations. NIST's risk framework supports a structured governance approach, but the institution must set its own delivery criteria and decision authority. Proposed targets belong in the plan; measured results belong in the report, with no interchange between them.

Make handover a testable deliverable

Before expanding, verify ownership of source updates, credentials, queue rules, monitoring, and incidents. Record the evaluated configuration and the changes that require retesting. A material change to a model, tool, permission, or source collection can affect behaviour and should trigger a proportionate review.

• Require evidence of the intended output and any external action, including its confirmed state.
• Test access boundaries, repeated inputs, unavailable dependencies, and partial completion.
• Have an operator pause, diagnose, recover, and resume the workflow using the documented process.
• Expand only when remaining risks, responsibilities, and acceptance decisions are explicit.