Notes / Workers
Before an AI Worker Touches Live Work: A Practical Evaluation Plan for Workers
A practical plan for deciding whether an AI Worker has earned a narrow, monitored role in a real business workflow.
By Rich Hill III. Published Aug 19, 2026. 13 min read.
An AI Worker can look ready long before it is ready for live work.
A polished demo may complete the happy path. A benchmark may show broad capability. Neither answers the operating questions that matter once the Worker can touch a customer record, trigger a handoff, update a system, or communicate outside the company: What exactly is it allowed to do? How will a team recognize a bad decision? Who can stop it? What happens when the workflow meets ambiguity, a changed system, or an exception?
The practical answer is to treat live enablement as a progressive authorization decision—not as a switch thrown after a promising demo. The Worker earns a limited role by producing evidence on the actual job, under known controls, with a recovery path that a real team can operate.
That approach is consistent with the NIST AI Risk Management Framework , which frames AI risk management around context, measurement, management, and governance across the lifecycle. It is not a universal checklist or a promise of safety. The details and tolerable failure modes must fit the workflow, the authority involved, and the consequence of an error.
What does “go live” actually mean?
“Go live” should not mean “the model can now do anything the demo showed.” It should mean that a named Worker has a specified role in a specified workflow, with a defined level of authority.
For one team, that first level might be reading a queue, assembling context, and drafting a proposed update for a person to approve. For another, it might be applying a routine internal status change after deterministic checks. It should not quietly include new tools, broader audiences, external commitments, or exceptional cases just because the model appears capable.
Write the authorization in plain language:
Frequently asked questions
Is a high evaluation score enough to approve live work?
No. A score can be one input, but the decision also depends on what cases were tested, how representative they are, the severity of errors, the authority granted, and whether access, approvals, stop conditions, logging, and recovery behaved as intended.
How large should the evaluation set be?
There is no universal number. Use enough normal, edge, misuse, and escalation cases to represent the proposed workflow and its meaningful variations. Keep a held-out portion, document what is absent, and make the set more demanding as the consequences of an error increase.
What should a human reviewer see before approving an action?
The reviewer should be able to see the proposed action, the authoritative context and current state, the evidence that the action is permitted, any uncertainty or policy conflict, and the simple decision they are being asked to make. A handoff that only says “please review” pushes the real work back onto the person.
Can a team use shadow mode instead of a live pilot?
Yes, when a Worker can observe or prepare recommendations without generating live side effects. Shadow mode is useful evidence, but it does not remove the need to evaluate control behavior and may not surface every condition of a real release. If a live pilot is considered, it should be narrow, observable, and backed by a practical pause and recovery path.
When should an AI Worker not go live?
Defer live authority when the job cannot be bounded, the relevant errors cannot be detected or recovered from, permissions cannot be scoped, a real escalation owner is missing, or representative testing cannot support the proposed action. A more limited role—or no automation for that step—can be the responsible decision.
Explore Workers