Model a real transaction, not an abstract assistant
Start with a concrete transaction: who makes the request, what information the assistant reads, which tools it may call, whose identity it uses, and where the result is written. Checking inventory, reading customer records, drafting a quotation, and changing payment terms may share one chat interface, but they do not carry the same risk. A useful threat model separates these operations instead of assigning one security rating to the entire assistant.
Map the trust boundaries around the user device, identity provider, model vendor, retrieval store, tool gateway, enterprise APIs, logs, and external network. At every boundary, ask whether data is encrypted, identity is verified again, authorization is reevaluated, and input can be controlled by another user or system. Inventory credentials, personal data, contracts, pricing, source code, and executable actions. The asset is not only information; it is also the organization's ability to perform a valid business transaction under a trusted identity.
Prioritize abuse paths introduced by the model
Conventional application threats still apply, but an assistant adds a probabilistic decision layer. Text in a document, email, webpage, or support message may be mistaken for an instruction, so reviewing only the user's prompt is insufficient. At minimum, exercise these paths:
- Direct and indirect prompt injection:Malicious text asks the model to ignore policy, reveal context, or invoke an unrelated tool. Retrieved content must be treated as data, never as trusted instructions.
- Privilege amplification:A shared, highly privileged service account lets ordinary users reach records or operations they could not access directly.
- Confused-deputy behavior:The assistant knows an action is generally valid but fails to verify that the current user, data scope, and business purpose belong together.
- Unsafe writes and replay:Retries, duplicate messages, and ambiguous confirmation can create duplicate orders, alter the wrong record, or send messages repeatedly.
- Data exfiltration:Sensitive content can leave its expected boundary through model requests, external links, tool arguments, debugging output, or overly complete conversation logs.
Rank scenarios using impact, detectability, and reversibility as well as likelihood. An incorrect lookup is usually easier to contain than an irreversible payment or a bulk notification. A quiet cross-department data leak may deserve stronger controls than a visible outage because it can continue without triggering an operational alarm.
Attach permission to the user and action, not the model
Prefer short-lived, scoped, attributable credentials and access internal systems as the current user whenever possible. If service identities are necessary, separate them by tool or workflow; do not let one administrative token span the CRM, ERP, and document store. Every tool call must pass backend authorization again. A model statement that the user has approval is not evidence of authorization.
Classify capabilities as read-only, reversible write, high-impact write, or administrative. Read access still needs field, row, tenant, and purpose restrictions. Reversible writes can create drafts. High-impact actions should display the exact target, changed fields, and expected consequence before requiring confirmation. That confirmation must be bound to specific parameters and an expiry time; a generic approval must not authorize later actions. If the system cannot clearly preview an action, limit its blast radius, or restore the prior state, the assistant should not execute it autonomously.
Enforce policy outside the model and test failure deliberately
The model can interpret intent and propose an action, but it should not be the final policy engine. A tool gateway should expose allowlisted operations with strict schemas, types, and value constraints while rejecting arbitrary SQL, URLs, or general-purpose commands. Backend controls should add data-scope checks, output filtering, rate limits, idempotency keys, network egress restrictions, and durable action audits. Logs should preserve who requested what, when policy was evaluated, and how the tool responded, while redacting tokens, sensitive prompts, and unnecessary document content.
Pre-production tests should include poisoned documents, cross-role questions, revoked users, expired credentials, malformed arguments, tool timeouts, partial completion, duplicate requests, and concurrent writes. The objective is not to prove that the model always reasons correctly. It is to prove that the gateway and system of record reject unsafe behavior when the model is wrong. Alerts also need named owners, and the team should rehearse disabling a tool, revoking credentials, and tracing one action from conversation to backend record.
A practical release decision comes down to three questions: Is the worst credible outcome bounded, will abnormal behavior be detected promptly, and can the action be stopped or reversed? If any answer is unclear, begin with read-only access, a limited dataset, or human approval. An experienced integration team can help turn those boundaries into controls that are testable, observable, and auditable.