Prompt Injection Prevention Checklist for RAG and Tool-Using Apps

Overview

This guide gives you a reusable prompt injection prevention checklist for RAG and tool-using apps. It is not a promise of perfect defense. Prompt injection is best treated as a class of risks that must be reduced across architecture, prompt design, retrieval, tool execution, output validation, logging, and evaluation.

The key operating assumption is simple: any text the model can read may contain adversarial instructions. That includes user messages, retrieved web pages, PDFs, support tickets, Slack exports, database rows, OCR output, code comments, and tool responses. In a modern LLM app development stack, those inputs often arrive through multiple hops, which makes it easy to forget where trust actually ends.

A useful mental model is to separate three things:

• Instructions: the policies and tasks your application intends the model to follow.
• Data: the content the model may analyze, summarize, classify, or transform.
• Actions: anything that changes state, sends data, spends money, triggers external APIs, or exposes secrets.

Most prompt injection failures happen when the application lets untrusted data act like instructions, or lets model output trigger high-impact actions without strong checks.

Use this article as a release checklist, an architecture review aid, and a regression-testing reference. It pairs well with operational disciplines like prompt versioning, structured output validation, and evaluation harnesses. For related implementation patterns, see Structured Output Prompting: JSON Schemas, Function Calling, and Parsing Reliability, Prompt Versioning Strategies: Git, Metadata, and Rollback Workflows, and How to Build a Prompt Evaluation Harness for Regression Testing.

Checklist by scenario

Start with the scenario that most closely matches your app. In practice, many systems need controls from more than one list.

1. Baseline checklist for any LLM application

• Assume all external text is untrusted. Do not treat retrieved content, uploaded files, or tool responses as safe simply because they come from an internal system.
• Keep system instructions separate from user and retrieved content. Your application code should preserve clear message roles and avoid concatenating everything into one undifferentiated prompt string.
• State limits explicitly. Tell the model that documents may contain irrelevant or malicious instructions and that it must treat them as content to analyze, not commands to obey.
• Minimize sensitive context. Do not include secrets, internal tokens, or unnecessary hidden instructions in prompts. If leaked or reflected, they increase damage.
• Prefer structured outputs. Use schemas for classifications, citations, tool arguments, and action requests instead of relying on free-form text. This narrows ambiguity and simplifies validation.
• Log raw inputs and decisions carefully. Keep enough telemetry for debugging, but redact secrets and personally sensitive fields.
• Build an evaluation set with adversarial cases. A prompt engineering guide is not enough; you need repeatable tests for known attack patterns.

2. RAG security checklist

• Tag retrieved content as untrusted. Whether you use vector search, keyword search, or hybrid retrieval, make the trust boundary explicit in code and prompt structure.
• Segment retrieved text from instructions. Use delimiters, metadata wrappers, or structured fields so the model can distinguish policy from source material.
• Filter or down-rank risky sources. Pages with hidden text, prompt-like phrasing, excessive directives, or suspicious formatting should be reviewed or excluded.
• Store source metadata with every chunk. You need document ID, origin, access tier, ingestion time, and sanitization status for investigation and rollback.
• Sanitize obvious hostile markup at ingestion. Remove or normalize hidden text, script fragments, malformed encoding, and junk instructions where appropriate, while preserving legitimate business content.
• Use retrieval rules that match authorization. A secure retriever should not surface documents the requesting user should not see.
• Require citations for factual answers. If your app answers from retrieved context, ask for evidence-backed responses tied to source IDs or snippets.
• Limit context size. Larger context windows can increase exposure to malicious content and make instruction conflicts harder to control.
• Review chunking strategy. Poor chunking can blend benign content with injected instructions, or separate warnings from the text they govern.
• Test indirect injection. Include cases where malicious instructions appear inside PDFs, docs, HTML comments, code blocks, or embedded text extracted from images.

3. Tool calling security checklist

• Classify tools by impact. Read-only search is different from sending email, creating tickets, deleting files, or initiating payments.
• Use allowlists, not broad tool access. Expose only the tools required for the user task and current step.
• Validate tool arguments server-side. Never trust model-generated parameters without type, range, enum, and format checks.
• Add user confirmation for high-impact actions. For destructive or external actions, require an explicit approval step that the model cannot silently bypass.
• Separate planning from execution. Let the model propose an action, but let application logic decide whether it is permitted.
• Constrain tool scopes. API keys and service accounts should have the minimum permissions necessary for each tool.
• Block secret access by default. The model should not be able to query hidden credentials, environment variables, or internal config unless there is a tightly controlled reason.
• Inspect tool outputs as untrusted input. A search result or API response can contain adversarial text just like a web page can.
• Rate-limit tool execution. This reduces abuse, runaway loops, and accidental cost spikes. For broader request-handling patterns, see API Rate Limit Handling for AI Applications.
• Keep an audit trail. Log who requested the action, what the model proposed, what was actually executed, and why it passed validation.

4. Multi-step agent and workflow checklist

• Bound the number of steps. A hard cap limits loops and repeated exposure to malicious content.
• Reset or summarize state carefully. Long chains can carry forward injected instructions if memory is not filtered.
• Define per-step permissions. Retrieval, reasoning, drafting, and execution should not all run under the same capabilities.
• Use step-specific prompts. A single general prompt for every task often creates broad, fragile behavior.
• Require deterministic gates between steps. Validate whether the step produced the expected schema, confidence signal, or evidence before proceeding.
• Escalate uncertain cases. If the model sees conflicting instructions, inaccessible sources, or suspicious directives, route to human review or a safer fallback.

5. Admin, internal knowledge, and enterprise app checklist

• Separate tenant and role boundaries. Retrieval and tools must respect account, role, and environment separation.
• Avoid mixing public and internal corpora by default. If you do mix them, label source trust and visibility clearly.
• Protect operational prompts. Store prompt versions, access permissions, and rollback paths with the same care you use for application configuration.
• Review logs for prompt leakage. Hidden instructions and confidential snippets can surface in traces if logging is too verbose.
• Define incident response steps. Know how to disable a tool, revoke a prompt version, purge a bad corpus segment, and reindex safely.

What to double-check

This section is the short list to review before deployment, after a prompt update, or whenever your workflow changes.

Trust boundaries

Can every piece of text in your system be labeled as trusted instruction, user input, retrieved content, or tool output? If not, your architecture may be hiding unsafe merges. This is especially common in apps that flatten prompts into one large string.

Authorization before retrieval and before action

Many teams check access when a user logs in, but not when the retriever fetches documents or when a tool acts on behalf of the user. Re-check both. Retrieval leaks and action leaks are separate failure modes.

Schema validation

If the model returns tool arguments, SQL fragments, JSON, or workflow decisions, are you validating them against strict schemas? This is one of the cleanest practical defenses because it converts vague model output into machine-checked contracts. If your implementation still depends on brittle text parsing, review structured output prompting patterns and reinforce server-side validation.

Output handling

Ask what happens after the model answers. Can it trigger an email, open a ticket, run a query, or write to a database? The safest prompt engineering in the world can still fail if downstream systems treat the answer as executable intent.

Ingestion pipeline hygiene

For RAG systems, security starts before retrieval. Double-check document parsing, OCR quality, metadata retention, deduplication, source ownership, sanitization, and deletion handling. A poisoned document that remains in the index will keep resurfacing.

Evaluation coverage

Your LLM evaluation set should include direct prompt injection, indirect injection through retrieved documents, malicious tool output, role confusion, data exfiltration attempts, and approval bypass attempts. For broader testing strategy, see LLM Evaluation Frameworks Compared.

Deployment model

Prompt injection defenses are partly architectural. If your app mixes retrieval, orchestration, model inference, and action execution in one service, operational isolation becomes harder. Review your deployment choices and failure domains. For infrastructure tradeoffs, see Serverless vs Containers for AI Inference.

Common mistakes

Most injection problems are not caused by a single bad prompt. They usually come from design shortcuts that feel harmless during prototyping.

• Relying on one warning sentence in the system prompt. Telling the model to ignore malicious instructions helps, but it should not be your only control.
• Letting the model call tools too freely. Tool use without server-side policy checks turns the model into a thin wrapper around privileged APIs.
• Treating internal data as trusted by default. Internal wikis, tickets, and shared docs can still contain hostile or misleading instructions.
• Skipping adversarial evaluation. If you only test happy paths, your app may look stable until a user uploads a poisoned document.
• Overstuffing context. Bigger prompts can reduce precision, raise costs, and increase the chance that unsafe instructions are included. If context growth is driving both spend and risk, revisit token discipline with cost control techniques for LLM apps.
• Parsing free-form output with regex alone. Regex can help with constrained tasks, but security-sensitive decisions deserve schemas, type checks, and execution guards. If you do use pattern matching, test it rigorously.
• No prompt version control. Without versioning, rollback is slow and incident analysis is fuzzy. Track prompt changes alongside deployment metadata.
• Logging everything. Full traces are useful until they expose secrets, private documents, or operational prompts. Balance observability with data minimization.
• Ignoring tool output injection. A search engine result, HTML scraper, or third-party API response can instruct the model just as effectively as a user message can.

A practical rule: if a piece of content can influence the model, assume it needs either separation, validation, or both.

When to revisit

Revisit this checklist whenever your app changes in a way that affects trust, retrieval, or actions. In operational terms, prompt injection defense is not a one-time prompt engineering task. It is a maintenance habit.

Review the checklist again:

• Before major planning cycles. Security debt tends to accumulate around roadmap changes, new data connectors, and new workflow automations.
• When prompts change. Even small wording edits can alter tool use, retrieval behavior, or output style. Pair changes with regression tests and prompt version metadata.
• When you add a new tool. Every new connector introduces permissions, argument validation, rate limits, and audit requirements.
• When your corpus changes. New document sources, OCR providers, ingestion scripts, and chunking strategies all affect RAG security.
• When model providers or settings change. A different model, context window, or tool-calling format can shift behavior in subtle ways.
• After incidents or near misses. Any suspicious output, unexpected tool proposal, or retrieval anomaly should feed back into tests.

To make this actionable, create a compact internal review routine:

1. List all current input sources and mark each one trusted or untrusted.
2. List all tools and assign impact levels: low, medium, high.
3. Confirm server-side validation exists for every tool argument and action request.
4. Run an adversarial evaluation suite before release.
5. Review logs and traces for accidental exposure of prompts, secrets, or restricted content.
6. Document rollback steps for prompts, retrieval indexes, and tool access.

If you want a repeatable operating model, combine this checklist with prompt versioning, structured output contracts, and a small regression harness. That gives you a practical defense loop: define trust boundaries, constrain actions, validate outputs, test adversarial cases, and revisit the controls whenever your workflows or tools change.

That loop is what makes prompt injection prevention sustainable for real AI deployment on cloud infrastructure. Not a single prompt. Not a single framework setting. A disciplined stack of controls that reduces risk before bad instructions reach your users or your systems.