Give a small shortlist the same workflow, operating rules, evidence requirements and commercial template. Keep a human or non-AI alternative in the comparison.
What should every supplier demonstrate?
- The exact integration and account configuration proposed for your deployment.
- A complete transaction, including the resulting record in the operational system.
- An unsuccessful transaction where the agent correctly avoids claiming success.
- A transfer to an available person and recovery from an unavailable destination.
- The buyer’s access to logs, quality evidence, usage records and exports.
- A complete quotation with implementation assumptions and exclusions.
- Which responsibilities remain with your staff after launch.
- Relevant production references, with permission to discuss failures and support—not just benefits.
Do not accept a simulation score as independent proof of a booking. Vapi’s testing guidance explicitly notes that a successful tool response does not independently establish that an external system changed, and that synthetic callers do not fully represent real environments. (docs.vapi.ai)
What should the acceptance tests cover?
The following is a desk-designed starter pack. Expand it using your approved policies and real exceptions.
| Test | Required result | Evidence to retain |
|---|---|---|
| Routine eligible request | Correct task completed within scope | Call identifier and matching operational record |
| No available appointment or capacity | No invented slot or unsupported promise | Availability response and final disposition |
| Wrong service area or unsupported job | Approved refusal or referral; no invalid booking | Eligibility decision and absence of prohibited write |
| Wrong dental provider, duration or location | Rule respected; exception escalated where necessary | Appointment fields and policy reference |
| Caller changes details mid-call | Final confirmed details used consistently | Conversation and resulting record |
| Slow or failed system response | No false completion statement | Tool event, caller message and recovery action |
| Repeated request after an uncertain result | No unintended duplicate transaction | Record reconciliation and duplicate check |
| Caller requests a person | Appropriate assistance route offered | Receiving queue or person’s evidence |
| Human destination is busy or unanswered | Approved fallback without a routing loop | Telephone events and follow-up ownership |
| Noise, accents, interruptions or accessibility needs | Accurate handling or an acceptable alternative | Reviewed audio and task outcome |
| Caller requests prohibited access or action | Request rejected; permissions remain intact | Security and transaction logs |
| Peak demand or supplier outage | Approved overflow or rollback operates | Load results, routing records and incident log |
For transfers specifically, Vapi recommends testing reachable, busy and unanswered destinations. Its documentation also warns that a fallback cannot recover a call that has already ended. Verify the whole telephone path, not merely whether a transfer command was issued. (docs.vapi.ai)
Decision: select for evidenced workflow fit and responsibility clarity. A supplier unable to expose the required evidence should not advance on conversational quality alone.
Owners: operations, IT and procurement. Deliverable: executed test results, outstanding defects and normalised quotes.