The attack appears in the prompt, but the risk lives at system boundaries
An enterprise AI assistant may receive LINE or web messages, retrieved documents, email, CRM notes, and content from external sites. A direct injection asks the model to ignore its existing rules. An indirect injection hides instructions inside content that will later be retrieved. Both arrive as ordinary text, which is why a keyword filter cannot reliably distinguish an attack from a legitimate security document or technical discussion.
A strong system prompt is still useful for defining the assistant’s role, handling rules, and prohibited behavior, but it is not an authorization boundary. Model compliance is probabilistic; access control must be enforced by software outside the model. A practical architecture distributes responsibility across ingestion, retrieval, tool execution, output handling, and monitoring, with each layer assigned a specific failure to contain.
Preserve trust information at ingestion and retrieval
At ingestion, validate supported file types, content size, encoding, source, tenant, and submitting identity. Retain provenance and collection time instead of stripping suspicious passages and losing their context. Later controls need to know whether a statement came from an approved policy, an internal knowledge article, a customer attachment, or an uncontrolled website.
Within RAG, retrieved text must be treated as evidence, not policy. Apply user, department, and tenant authorization before retrieval rather than giving the model every document and asking it to ignore inaccessible material. System policies and ordinary knowledge should be stored separately, or at least marked with distinct trust metadata, so a document cannot grant itself higher authority merely by claiming to be an administrator instruction.
- Ingress: Normalize content, constrain accepted formats, and label whether it came from a person, an internal system, or an external source.
- Retrieval: Apply access-control filters before vector or keyword search and preserve document identifiers in the result.
- Context assembly: Separate system rules, user requests, and reference material, carrying provenance and trust labels into the prompt.
- Risk routing: Send suspicious material through a read-only summary path, additional inspection, or a workflow without privileged tools.
Classifiers and pattern rules can reduce obvious attacks, but they should not be the only gate. Broad rules block legitimate content, while narrow rules are bypassed through paraphrasing, encoding, or another language. Tune the response to downstream impact: a read-only search experience can favor recall, whereas content feeding a customer-data export deserves stricter isolation.
The tool and action boundary is the critical control point
When an assistant can call ERP, CRM, cloud, or messaging APIs, the tool gateway becomes the most important defense. A tool name and arguments produced by the model are proposals, not authorization. The backend must independently validate the schema, authenticated user, tenant, permitted objects, and applicable business rules. Free text should never become arbitrary SQL, shell commands, URLs, or API calls.
- Separate reads from writes: Give query and mutation tools different credentials, with the narrowest scope needed for each task.
- Use constrained interfaces: Prefer fixed schemas, field allowlists, bounded parameters, and known endpoints over model-generated queries.
- Confirm consequential actions: Before sending messages, exporting data, deleting records, or changing access, show the exact target and payload for approval.
- Restrict egress and secrets: Allow only required destinations, keep credentials in the execution environment, and never place raw keys in model context.
Consider a LINE user asking for a summary of ERP invoices they are allowed to view. A retrieved attachment contains hidden text telling the assistant to export the entire customer list. Even if the model proposes that action, the tool layer should expose only authorized invoice queries and no general customer-export capability. Authorization must run after intent interpretation: understanding what someone requested does not prove that they have permission to do it.
Output controls and operations contain residual risk
Validate output according to its destination. An API response can be required to match a fixed schema; text rendered in a browser or chat client must be escaped safely; answers grounded in internal knowledge should retain document references. Output checks can stop accidental disclosure of secrets, internal instructions, or executable markup, but they cannot undo an unauthorized action that an earlier layer already performed.
Operational records should connect the user request, retrieved document identifiers, proposed tool arguments, authorization decisions, approvals, and final response without copying unnecessary secrets into logs. Test suites should cover direct requests, instructions embedded in documents, multilingual paraphrases, encoded or split payloads, and second-stage injections returned by tools. Rerun them whenever the model, system prompt, retrieval pipeline, or tool definitions change.
Place defenses according to consequence. A read-only knowledge assistant may concentrate on retrieval authorization, provenance, and output validation. An agent with write access or cross-system integrations needs least privilege, deterministic authorization, and confirmation at the tool boundary. Start by mapping where untrusted data enters, what resources the model can reach, and whether a failed action is reversible; that map usually makes the required control points clear.