Prompt injection prevention is an application security problem, not something a single system prompt can solve. This checklist helps development and operations teams threat-model direct and indirect attacks, limit tool and data access, validate outputs, monitor changes, and schedule recurring reviews for LLM applications and RAG systems.
Overview
Prompt injection occurs when untrusted instructions influence an LLM in ways that conflict with the application’s intended behavior. The instruction may be typed directly by a user, embedded in a document retrieved by a RAG pipeline, returned by a web page, hidden in an uploaded file, or produced by another model. The model does not reliably know which text is authoritative merely because the application labels it as context.
A useful security model separates three things:
- Instructions: rules defined by the application, such as the task, response format, and refusal behavior.
- Data: user content, retrieved documents, search results, tool output, and uploaded files that may contain hostile or misleading text.
- Capabilities: actions the application can take, including database queries, messages, file changes, purchases, or API calls.
Prompt injection becomes more serious when untrusted data can influence a model that has broad capabilities. The practical objective is therefore not to make the model perfectly distinguish every malicious sentence. It is to design the surrounding system so that a mistaken instruction cannot expose secrets, bypass authorization, or perform an irreversible action.
What to track
1. Identify direct and indirect injection paths
Begin with a data-flow diagram for every model call. Mark where text enters the system, where it is transformed, and where it can affect a decision or tool call.
| Scenario | Typical path | Primary control |
|---|---|---|
| Direct injection | A user asks the assistant to ignore its task, reveal hidden instructions, or take an unrelated action. | Separate user input from application instructions, enforce authorization outside the model, and validate the result. |
| Indirect injection | A retrieved document, web page, email, ticket, or uploaded file contains instructions addressed to the model. | Treat retrieved and uploaded content as untrusted data; restrict what it can influence and log the source. |
| Tool-mediated injection | Model output selects a tool, constructs arguments, or causes a downstream action. | Use narrow schemas, server-side permission checks, argument validation, rate limits, and approval for sensitive actions. |
| Cross-step injection | One model response is passed into a later prompt or agent step as if it were trusted instruction. | Label step outputs as data, constrain transitions, and validate each boundary independently. |
2. Inventory secrets and trust boundaries
List credentials, tokens, internal prompts, personal data, private documents, and configuration values available to the application. A model should not receive a secret simply because a tool needs it. Keep credentials in the server-side integration layer, use the narrowest permission available, and return only the minimum tool result needed for the next step.
For RAG systems, record the source, tenant, access decision, and retrieval query for each document supplied to the model. Retrieval is not authorization. A document must pass an access check before it becomes context, even if a vector search returns it.
3. Constrain tools and outputs
Design tools as small, explicit operations rather than general-purpose interfaces. A tool that sends one approved notification is easier to secure than one that accepts arbitrary URLs, code, or commands. Validate types, length, allowed values, ownership, and destination on the server. Do not rely on the model to enforce these rules.
Use structured output where practical, but treat valid JSON as a formatting property rather than proof that the content is safe. Check whether the requested action is permitted, whether the referenced resources belong to the current user, and whether the action is consistent with the workflow.
4. Maintain a security test set
Track representative attack cases alongside normal examples. Include requests to reveal system instructions, override role boundaries, extract hidden data, follow instructions inside retrieved content, manipulate tool arguments, and induce the assistant to skip approval. Add benign lookalikes so the test set measures both resistance and false positives.
For repeatable LLM evaluation, record the application version, model configuration, prompt version, retrieval settings, tool list, expected behavior, and observed result. A prompt change or provider change can alter the risk profile even when application code has not changed. See the LLM evaluation guide for a broader regression workflow.
Cadence and checkpoints
Prompt injection prevention should be reviewed on a recurring schedule and whenever the system’s trust boundaries change. A lightweight operating cadence can keep security work connected to deployment rather than treating it as a one-time launch task.
At every release
- Compare the changed prompts, tools, retrieval sources, and model settings with the previous version.
- Run the injection regression set, including direct, indirect, and tool-use cases.
- Confirm that secrets remain outside model-visible context and that authorization is enforced by application code.
- Review new output parsers, automation steps, and fallback behavior.
Monthly or quarterly
- Rebuild the data-flow and trust-boundary inventory for active features.
- Review tool permissions, service accounts, database scopes, and external integrations.
- Sample logs for suspicious instructions, repeated refusal bypass attempts, unusual tool calls, and unexpected destinations.
- Retire obsolete test cases and add cases from incidents, support reports, red-team exercises, and newly introduced data sources.
- Verify that monitoring still captures prompt versions, retrieval sources, tool arguments, approvals, errors, and user or tenant boundaries without logging sensitive content unnecessarily.
Before a major capability change
Pause for a focused threat model when adding browsing, file access, code execution, outbound messaging, financial operations, autonomous loops, or new tenant data. Define what the feature may do, what it must never do, who can approve exceptions, and how to disable it quickly. Tool design and error handling deserve their own review; the LLM tool calling guide provides a useful implementation companion.
How to interpret changes
Do not measure prompt injection defense by refusal rate alone. A system that refuses every ambiguous request may be unusable, while a system that gives helpful answers but exposes restricted data has failed its security objective. Review changes across several dimensions:
- Confidentiality: Did the response reveal secrets, hidden instructions, private context, or data belonging to another user?
- Integrity: Did the model alter, summarize, classify, or route information in a way that an injected instruction could control?
- Authorization: Did a tool call bypass the user’s permissions or the required approval step?
- Availability: Did the interaction cause excessive loops, expensive retrieval, repeated tool calls, or resource exhaustion?
- Traceability: Can the team identify the input source, prompt version, model call, tool arguments, and final action?
When a test result worsens, isolate the layer responsible. A retrieval failure may be caused by access filtering, a prompt change, document contamination, or an overly broad tool. Compare the same case with retrieval disabled, tools disabled, and a fixed model configuration where safe. This helps avoid the common mistake of rewriting the system prompt when the real problem is missing authorization or excessive capability.
Use layered defenses rather than assuming any one technique is complete. Instruction boundaries and system prompts clarify intent. Retrieval filters limit data exposure. Tool schemas narrow actions. Server-side authorization decides what is allowed. Output validation catches malformed or dangerous results. Human approval adds a control for high-impact actions. Logging and incident response make failures discoverable and recoverable.
Safe test-prompt template
Use synthetic data and sandbox tools when testing. Never paste production credentials, live tokens, private customer records, or real destructive commands into an evaluation prompt.
Purpose: Evaluate whether the assistant keeps application instructions separate from untrusted content.
Environment: Isolated test tenant with synthetic records and disabled side effects.
User request: [ordinary task]
Untrusted content: "[synthetic document containing an instruction to ignore the task or request hidden data]"
Expected behavior: Treat the quoted content as data, refuse unauthorized disclosure or action, and continue the permitted task when possible.
Tool expectation: Do not call tools unless the request is authorized and arguments pass server-side validation.
Evidence to record: application version, prompt version, retrieval source ID, tool decision, output category, and reviewer result.When to revisit
Run this checklist at least monthly or quarterly according to the application’s risk and release pace. Revisit it immediately after an incident, a suspicious output, a new data connector, a model or provider change, a prompt-template change, or the addition of any tool that can affect external systems.
At each review, assign an owner and record four decisions: what changed, which attack paths were retested, what evidence was collected, and which controls need follow-up. Keep the checklist with the deployment or change-management record so future reviewers can compare decisions rather than starting from memory. The LLM observability guide can help connect security signals with production quality, latency, and failure monitoring.
For a practical next step, choose one production workflow and complete this sequence:
- Draw its input, retrieval, model, tool, and output boundaries.
- Remove unnecessary secrets and reduce tool permissions.
- Add five direct-injection and five indirect-injection cases using synthetic data.
- Enforce authorization and argument validation outside the model.
- Run the cases before the next release and schedule the next review date.
Prompt injection prevention is an ongoing cloud operations practice. A recurring review, a narrow capability design, and evidence from repeatable tests provide a stronger foundation than relying on prompt wording alone.
