Proposed lead: Allison Koenecke, with accessibility and customer-quality representatives.
Why it matters: Koenecke and colleagues’ research on auditing speech recognition for people with aphasia examines shortcomings in audit practice. It provides a basis for questioning evaluation design—not for declaring that a particular current voice-agent supplier has failed. (arxiv.org)
Question to put to the expert:
How should a buyer determine whether an acceptable overall completion rate conceals poor service for particular accents, languages, speech patterns or call conditions?
A strong answer should cover:
- A caller and environment test plan grounded in the population served.
- Task completion, effort and escalation outcomes alongside recognition accuracy.
- Human assessment involving relevant users, rather than synthetic accents alone.
- Separate reporting of sparse samples and uncertainty.
- A privacy-conscious way to study differences without casually inferring sensitive identities from voices.
- A usable alternative when the system repeatedly misunderstands someone.
Evidence to request: A stratified evaluation plan, subgroup sample counts, error examples and remediation ownership.
Follow-up: Which apparently reassuring aggregate result would you refuse to publish?
Expert quote: [Reserved for an approved interview quotation.]