Progress from synthetic testing to controlled telephone tests, then limited live traffic with a named fallback owner. Treat time, spending, traffic exposure and permissions as separate limits.
NIST’s AI RMF calls for documented test sets and metrics, evaluation under conditions similar to deployment, and ongoing tracking of risks. It does not establish a universal number of calls that makes a voice-agent pilot safe. (airc.nist.gov)
What must the pilot charter specify?
- One workflow, named locations, operating hours and supported caller groups.
- Maximum spending, traffic allocation and concurrent workload.
- Allowed actions and prohibited actions.
- The baseline or comparison group and how calls are allocated.
- Success definitions, exclusions, minimum evidence and decision thresholds.
- Reviewers who can inspect outcomes independently of supplier scoring.
- Immediate pause conditions and the person authorised to activate them.
- A tested route back to the previous service.
- The observation window for corrections, repeat contact and downstream completion.
Where practical, compare contemporaneous groups with similar call types, hours and locations. Record any changes in staffing, advertising, seasonality or capacity rather than attributing the entire difference to AI. This is our recommended evaluation design, not a claim that every buyer can run a randomised trial.
Which measures belong on the pilot scorecard?
Use the following definitions consistently across suppliers and the baseline.
| Measure | Recommended definition | What not to substitute |
|---|---|---|
| Autonomous completion | Unique eligible tasks completed correctly without human rescue ÷ unique eligible tasks entering the pilot | Calls that avoided transfer |
| Overall workflow completion | Unique correct completions, including human rescue, ÷ the same eligible-task population | Supplier-labelled “successful calls” without verification |
| Booking integrity | Correct appointments ÷ appointments created | Spoken booking confirmations |
| Live handoff success | Required/requested live handoffs accepted by a suitable staffed destination ÷ such handoffs | Transfer commands or ringing destinations |
| Callback fulfilment | Promised callbacks completed within the approved commitment ÷ promised callbacks | Callback tickets merely created |
| Rework and repeat contact | Corrections and related repeat contacts within the predefined observation window | Only complaints submitted formally |
| Fully loaded outcome cost | All relevant workflow costs ÷ unique correct completions | AI cost per minute |
| Critical incidents | Count and severity of prohibited actions, disclosures or unsafe handling | An average quality score |
Track unexpected and out-of-scope calls separately; do not silently remove failures from the denominator. Review results by location, intent, language and relevant audio conditions, showing small sample sizes rather than presenting unstable percentages as established performance.
What should make you stop, repair or expand?
Recommended immediate pause triggers: unauthorised disclosure or action; a materially false completion claim; a broken critical escalation path; or loss of the ability to monitor or revert the affected workflow. Investigate, reconcile affected records and require retesting before restart.
Repair and retest: correctable defects remain within a scope that can be safely isolated.
Extend: evidence is inconclusive because volume, important scenarios or downstream outcomes are insufficient. A calendar deadline is not evidence of success.
Expand: all hard controls pass, agreed operational and economic thresholds are met, and evidence covers the intended next population. Zero observed critical incidents should remain a launch condition, not be presented as proof of zero underlying risk.
After-hours rule: do not promise an immediate human transfer to an unstaffed queue. Use an approved on-call route or clearly described callback arrangement; exclude workflows whose urgency that arrangement cannot support.
Owners: operations and quality, with risk and finance sign-off. Deliverable: a written expand, repair, extend or stop decision.