They must be tested under those conditions; a fluent demonstration does not establish performance across your callers. Evaluate task completion and caller effort, not just transcription quality or how natural the voice sounds.
The 2026 τ-Voice research benchmark evaluates full-duplex agents across 278 tasks and reports lower task performance under its realistic audio conditions than under clean conditions. Those results describe its models, tasks and experimental setup—not the completion rate of every commercial receptionist. (arxiv.org)
Our recommended test groups include:
- Accents and languages relevant to the actual caller population.
- Street noise, speakerphone and degraded connections.
- Callers who interrupt or correct a previous answer.
- Long pauses, hesitation and slower speech.
- Names, addresses and reference numbers that require exact capture.
- Requests for a person or an alternative communication channel.
Measure how often callers must repeat themselves and whether corrections reach the final record.
For latency, distinguish the time to a first acknowledgement from the time to a useful answer and the time to complete a system action. Review median and slower-tail performance rather than one average.
Do not adopt a universal “under 500 milliseconds” procurement rule without testing the whole interaction. Set workload-specific limits for silence, interruption recovery and task completion, then verify them on the intended call route.