Release limited traffic when the responsible team is available, then expand on evidence. Do not make the first production experiment an unsupported overnight launch.
An illustrative rollout sequence is:
- Internal telephone calls and approved external testers.
- A controlled traffic slice, perhaps 5–10% of the eligible route.
- A larger slice, perhaps 25%, after review.
- Broader coverage only after the agreed acceptance gates hold.
Those percentages are planning examples, not universal best practice. At low volumes, a defined location or time window may be more practical.
What should be checked during the first operating days?
Proposed initial controls:
- Review every one of the first 50 live interactions, where authorised.
- Review every failed transfer, transaction anomaly and complaint.
- Reconcile every claimed booking against the authoritative system.
- Sample routine successes by location, call type and operating period.
- Keep a daily log of changes and incidents.
- Freeze versions between formal comparisons.
Do not rely solely on a supplier-generated success label. Vapi’s structured outputs are extracted from conversations; use them to support triage, while verifying transactional claims against the system of record. (docs.vapi.ai)
What belongs in the outcome ledger?
| Measure | Proposed definition |
|---|---|
| Eligible tasks offered | Unique tasks meeting the predeclared pilot scope |
| Autonomous completion | Correct, policy-compliant task completed without human rescue |
| Human-rescued completion | Correct final outcome requiring a person after AI involvement |
| Unresolved task | No verified correct completion within the defined outcome window |
| Booking integrity | Required fields and permissions correct in the authoritative record |
| Accepted human handoff | Intended human destination accepts the transfer under the defined rule |
| Repeat contact | Further contact concerning the same task within the chosen window |
| Caller effort | Questions repeated, unsuccessful transfers and avoidable recontacts |
| Fully loaded outcome cost | Attributable workflow cost divided by unique verified completions |
Do not compare only an easy AI queue with a difficult human queue. Prefer random assignment where feasible, or a documented matched comparison controlling for location, call reason and operating period. For home services, note weather, staffing and available capacity when interpreting booking differences.
How do you size concurrent capacity?
Start with observed arrival patterns and call lengths, then load-test the intended peak and fallback behaviour.
For illustration, 120 calls per hour averaging four minutes imply eight simultaneous calls on average: 120 × 4 ÷ 60. That is not a peak-capacity requirement; bursts and long calls require separate testing.
Stop immediately for the predefined critical incidents. Do not wait for a weekly dashboard if the agent is confirming nonexistent appointments or exposing another customer’s information.