Constrain its authority and validate actions outside the conversation model. Our recommendation is to use prompts for behaviour, but enforce consequential permissions and transaction rules in the connected application.
NIST identifies confidently false or internally inconsistent output as a generative-AI risk. That is why a natural-sounding explanation should not substitute for an authorised source or verified action. (nvlpubs.nist.gov)
Require these controls:
- Approved knowledge: Identify the source and owner of prices, policies and opening hours.
- Narrow permissions: Permit only the records and actions required by the workflow.
- Input validation: Reject invalid locations, appointment types and unsupported values.
- Action confirmation: Require appropriate caller confirmation before the change.
- Safe error handling: Do not turn timeouts or ambiguous results into claims of success.
- Duplicate protection: Define how repeated requests and retries are reconciled.
- Audit evidence: Preserve the relevant action, result and configuration identifiers.
Check the underlying product’s permission model rather than assuming it matches employee permissions. Open Dental explicitly states that its API security permissions are organised differently from those in the main application. (opendental.com)
Include adversarial tests such as a caller asking the agent to ignore policy, change an unauthorised record or disclose another customer’s information. Passing these tests is an acceptance requirement—not a guarantee against every future failure.