Proposed lead: Allison Koenecke and Kwindla Hultman Kramer.
Why it matters: LiveKit distinguishes scripted tests, whole-conversation simulations and observation of live sessions. NIST’s generative-AI profile also treats pre-deployment testing as part of broader risk management, rather than a complete guarantee of deployed behaviour. (docs.livekit.io)
Question to put to the expert:
If an AI system grades another AI system’s calls, how do we know the score measures correct outcomes rather than plausible conversation?
A strong answer should cover:
- Business-approved labels and independently reviewed examples.
- Deterministic checks of transaction state where possible.
- Calibration of automated judgements against human review.
- Separate error analysis for false passes and false failures.
- A withheld evaluation set, not just scenarios used to tune the agent.
- Audio testing, live observation and a route for customer complaints to become new tests.
Ask how the pilot’s sample size follows from the decisions being made and the errors the buyer needs to detect. A fixed test count should not be presented as universal assurance.
Evidence to request: A redacted labelled sample, disagreement log, evaluator version and rationale for the evaluation sample.
Follow-up: Show a call the evaluator marked successful that an operator rejected—and explain the correction.
Expert quote: [Reserved for an approved interview quotation.]