Use vendor testing to accelerate development, then verify the evaluator and the operational outcome. ElevenLabs documents simulation, next-response and tool-call tests. The EVA-Bench research framework separately evaluates task accuracy and conversational experience, including accent and noise perturbations. Neither establishes how a named supplier will perform on your particular calls. (elevenlabs.io)
A useful evidence hierarchy is:
- Feature page: a capability is described.
- Demonstration: it worked in the shown example.
- Buyer-controlled test: it worked against known rules and expected system states.
- Controlled live pilot: it worked on a defined share of real demand with monitored exceptions.
- Ongoing operating evidence: results remain acceptable after changes and under changing call conditions.
NIST’s March 2026 monitoring report emphasises why pre-deployment evaluation needs to be complemented by observation after deployment: controlled tests do not capture every real-world input or unforeseen output. (nist.gov)
Which metrics should decide an AI call center comparison?
These are the desk’s proposed definitions. Put the definitions—not merely the metric names—into the trial agreement.
| Measure | Proposed calculation or evidence | What it prevents |
|---|---|---|
| Verified task completion | Unique correctly completed, policy-compliant tasks ÷ unique eligible tasks | Treating a pleasant conversation as a transaction |
| Autonomous completion | Verified completions requiring no human work | Hiding rescue labour inside an automation claim |
| Human-rescued completion | Verified completions that required a person | Counting a successful hybrid workflow as autonomous |
| False confirmation | Calls claiming success without the required final system state | Accepting “you are booked” without a booking |
| Required-handoff success | Correctly completed handoffs ÷ all cases that required one | Reporting only transfers the agent chose to attempt |
| Repeat contact | Related repeat attempts within a pre-agreed, workflow-specific window | Inflating success by counting repeated unresolved calls |
| Record integrity | Correct fields, no unwanted duplicates, authorised changes only | Measuring conversational quality while ignoring system damage |
| Caller effort and timing | Repetition, interruption recovery, abandonment, and median/tail response timing | Optimising only average latency or call length |
| Performance by test condition | Outcomes by workflow, language, accent/noise condition and load | Allowing an acceptable average to conceal failure pockets |
| Fully loaded economics | Total attributable cost ÷ unique verified outcomes | Comparing incomplete usage bills |
Do not infer sensitive demographic characteristics from voices to populate a dashboard. Use consented or synthetic test conditions and an approved evaluation design.
What should the pilot sequence look like?
Our recommended sequence is:
1. Define the baseline and the permitted work. Select one workflow, establish its current completion and cost measures, and document prohibited actions. Keep the incumbent process or another credible alternative in the comparison.
2. Run deterministic integration tests. Exercise availability reads, writes, retries, duplicate prevention, authorisation and recovery. Check the resulting records independently of the conversation summary.
3. Run repeated end-to-end calls. A 25-case synthetic screening pack can be a useful start, but it is not statistical assurance. Repeat critical cases with changed phrasing, interruptions, noise, unavailable slots and dependency failures.
4. Run a limited live comparison. Where feasible, assign comparable eligible demand across alternatives. Record differences in hours, locations, call mix, configuration and staffing so that an easier workload is not mistaken for a better supplier.
5. Review outcomes after the call. Reconcile records and repeat contact, calibrate automated assessments against human review, and investigate both false passes and false failures.
6. Continue monitoring after approval. Retest material changes and retain an immediate rollback route. NIST’s AI RMF explicitly includes post-deployment monitoring, override, incident response, recovery and change management. (airc.nist.gov)
Should procurement use a weighted vendor score?
Yes—but only after pass/fail gates. The following weights are proposed, not empirically validated.
| Criterion | Suggested weight |
|---|---|
| Verified task completion and record correctness | 30% |
| Safety, privacy and permission controls | 25% |
| Human handoff and recovery | 15% |
| Fully loaded economics | 15% |
| Monitoring, support and change control | 10% |
| Portability and exit | 5% |
Score each item from 1 to 5 using agreed evidence anchors. An unresolved legal basis, an unauthorised system action, a critical false confirmation or an unusable required fallback should block approval rather than be averaged away. Zero critical failures observed in a small test is a gate result, not proof of zero production risk.