How to secure an LLM application: risks and defenses

Every security model you have ever used rests on a boundary. This is code and that is data. This input is trusted and that one is not. Injection attacks in traditional software are exactly the failure of that boundary, and the fixes work by restoring it, through parameterised queries and escaping and type systems that keep the two apart. Language models break this in a way that has no clean fix. A model has no reliable boundary between instructions and data, because both reach it as one undifferentiated stream of text it cannot tell apart, so the model itself can never be made trustworthy, and securing an application built on one means re-imposing that boundary outside the model: constraining what it can reach through least privilege enforced in infrastructure, validating what it emits with deterministic code rather than another model, and requiring human approval wherever an action cannot be undone.

This guide covers the defenses that follow from that, in the order they matter, the risks worth knowing by name, and the popular measures that provide less protection than they appear to. It is written for someone shipping a system rather than studying attacks, and it deliberately stays at the level of architecture rather than technique.

Why LLM security is different

The traditional web security list is familiar: injection, broken authentication, misconfiguration. Those risks still apply, because an LLM application is still an application. But building on a model adds a category of risk with no real precedent, and importing habits from web security alone will leave the new surface uncovered.

The difference comes from what the model does. It takes text and produces text, and it treats all text it receives as material to act on. A system prompt, a user message, the contents of a retrieved document, the body of an email it was asked to summarise, the text on a web page it fetched: to the model these arrive in the same channel, differing only in position and phrasing. That is the whole problem in one sentence. When an attacker can get text in front of the model, they are speaking to it with something close to the same authority as the developer who wrote the system prompt.

This is why prompt injection is not merely the top item on the risk lists but a different kind of item. It is not a bug in an implementation that a patch removes. It exploits how these systems work, which means the honest goal is not preventing it but limiting what a successful injection can accomplish.

The boundary that does not exist

Sit with the consequence for a moment, because it reorders your priorities.

If you cannot stop untrusted text from being interpreted as instruction, then no amount of instructing the model to resist will be sufficient. Telling it to ignore attempts to change its instructions helps at the margin and fails against inputs the phrasing did not anticipate, which is the same shallowness that makes jailbreaking persistent. Defenses that live inside the prompt are advisory, and advisory controls are not security controls.

What follows is the organising rule for everything below: security controls must be enforced somewhere the model cannot influence. If a rule exists only as a sentence in the system prompt, the model can be talked out of it. If the rule exists as a permission the model was never granted, there is nothing to talk out of. Every effective defense below is a version of moving a control from the first category into the second.

Least privilege, enforced in infrastructure

The highest-value control by a wide margin is limiting what the model can reach, because it caps the damage of every other failure at once.

In practice this means the set of tools available to a model is an explicit allowlist defined in your code and enforced by your infrastructure, not a list described to the model in text. It means credentials scoped to the narrowest possible permission rather than a general-purpose key, short-lived rather than durable, and separate per task rather than shared. It means the model has read access where reading is all it needs, and write access only where writing is the point. It means sensitive logic lives in application code that the model calls, rather than in reasoning the model performs, so that the decision to permit an action is made by a program rather than by a system that can be persuaded.

There is a further refinement worth adopting deliberately, because it addresses the specific shape of the risk: an agent should not hold powerful tools in the same step in which it consumes untrusted content. If a process reads an external document and can also send email or write to a database in that same turn, then anyone who can place text in that document has a path to those capabilities. Separating those phases, so that reading untrusted material and taking consequential action never occur with the same permissions active, removes the path rather than filtering it.

Treat model output as untrusted input

The second principle inverts the usual framing. Most discussion focuses on what goes into the model. Just as much damage comes from what comes out of it being trusted downstream.

Whatever a model produces should be treated exactly as you would treat a string submitted by an anonymous user on the internet, because functionally that is what it is. If model output becomes a database query, parameterise it. If it becomes part of a web page, encode it. If it becomes a shell command or a file path, validate it against a strict allowlist. If it becomes an argument to a tool call, check the argument against expected types and ranges before executing. The category of failure here is easy to miss because the output looks like it came from your own system.

One point deserves emphasis because it is a common mistake: do not use a language model to validate a language model's output where correctness matters. A second model is subject to the same manipulations as the first, so a validator built from the same material inherits the weakness it is supposed to catch. Use deterministic code for anything you actually need to be certain about, and reserve model-based checking for things that cannot be expressed in code, such as tone or topical relevance, where a probabilistic answer is acceptable.

Human approval for irreversible actions

The third principle is the cheapest insurance available. Sort the actions your system can take by whether they can be undone, and require a person to confirm the ones that cannot.

Sending an external message, deleting data, moving money, publishing content, changing permissions, and calling any third-party interface with write access all belong in this category. Reading, searching, drafting, and summarising generally do not. A human-in-the-loop gate on the irreversible subset converts a class of catastrophic failures into a class of annoying ones, and it does so without depending on the model behaving correctly at all.

This is worth defending against the objection that it undermines automation. It does reduce autonomy, deliberately, and the appropriate amount of autonomy is a function of how bad the worst case is. A system that drafts replies for review and one that sends them are different products with different risk profiles, and the second should be chosen knowingly rather than by default. Many of the agent failures that become incidents are cases where an approval gate was absent on an action that could not be walked back.

Segregate and label untrusted content

The fourth principle mitigates what it cannot prevent. Since untrusted text will reach the model, structure the context so that its status is at least marked, and so that your own pipeline can reason about it even though the model cannot fully be relied upon to.

Practically, this means keeping retrieved documents, tool results, and user-supplied content in clearly delimited regions of the prompt rather than concatenated into instructions, so that your application knows which spans are untrusted even if the model treats them uniformly. It means applying access control to your retrieval corpus, so that a RAG system cannot surface documents the current user should not see, which is a data-leak risk that is easy to overlook because the retrieval layer feels like infrastructure rather than an authorisation boundary. It means inspecting retrieved content before it enters context, since a document that contains instruction-shaped text is at minimum worth flagging. And it means never placing secrets in a system prompt, because a system prompt should be assumed to be discoverable rather than confidential.

These measures reduce exposure rather than eliminating it. They belong in the stack, and they are not a substitute for the permission controls above.

Contain the blast radius

The fifth principle assumes something will go wrong and limits what that costs.

Run tool execution in a sandbox with no ambient credentials and no network access it does not need, so that code the model generates cannot reach further than intended. Set rate limits and hard cost ceilings per task and per user, with automatic circuit breakers, because an agent operating in a loop can consume resources at a rate that becomes a financial incident rather than merely a performance one. Log every action with enough context to reconstruct the path from input to consequence afterwards, since without that trace an investigation has nothing to work with. And test adversarially on a schedule rather than once, treating red-teaming as ongoing practice, because the threat surface changes with every capability you add.

The risks worth knowing by name

The industry reference here is the OWASP list for large language model applications, and knowing its categories is useful shorthand even if you do not follow it formally. The headline risk is prompt injection, both direct from a user and indirect through content the system retrieves. Sensitive information disclosure covers data escaping through responses, logs, caches, embeddings, and integrations, which is a wider surface than most teams check. Supply chain and data poisoning cover compromised models, packages, and training or retrieval corpora. Improper output handling is the downstream trust problem described above. Excessive agency is the permission problem, and it is where least privilege applies. System prompt leakage is why prompts should hold no secrets. Vector and embedding weaknesses cover access control in retrieval systems. Misinformation covers the model asserting false things confidently, which overlaps with hallucination mitigation and is a reliability risk as much as a security one. Unbounded consumption covers resource and cost exhaustion.

For agent systems specifically, two additional patterns matter: an agent's objective being redirected by text it encounters, and an authorised tool being used in a destructive way with arguments the system never validated. Both are addressed by the same controls, permissions and argument validation and approval gates, rather than by better instructions.

What provides less protection than it appears

Several popular measures are worth keeping while understanding their limits. Instructing the model to refuse manipulation is advisory and defeated by phrasings the instruction did not anticipate. Input filters that match known injection patterns catch unsophisticated attempts and miss novel ones, since the space of natural language cannot be enumerated. Fine-tuning to resist manipulation raises the bar without closing the gap, for the same reason safety training can be circumvented. Using a second model as a judge inherits the first model's weaknesses. And the assumption that a more capable model will be more secure does not hold, since greater capability tends to mean broader tool access and more consequential actions rather than better resistance.

None of these are worthless, and defense in depth means keeping cheap partial measures. The mistake is treating any of them as the control that makes the system safe, when each is a filter rather than a boundary.

Where to start

If you are securing an existing system, a short sequence gets most of the value. Enumerate every tool and credential the model can reach and remove everything not required for the current task. Move any control that exists only as an instruction into code. Identify every action that cannot be undone and put an approval gate in front of it. Treat all model output as untrusted before it reaches another system. Confirm your retrieval layer enforces the same access control as the rest of your application. Add cost ceilings and logging. Then test adversarially, and repeat that test whenever you add a capability, since new tools are what change the risk profile.

The short version

Securing an application built on a language model differs from ordinary application security because the model has no boundary between instructions and data: system prompts, user input, retrieved documents, and tool results all arrive as one stream of text, so anyone who can place text in front of the model is speaking to it with something close to developer authority. Prompt injection therefore cannot be patched away, and the goal is limiting what a successful injection can do. The organising rule is that controls must be enforced where the model cannot influence them, which makes an instruction in a system prompt advisory rather than a security control. In order of value: apply least privilege to tools and credentials, enforced by infrastructure rather than described in text, and avoid granting powerful capabilities in the same step that consumes untrusted content; treat model output as untrusted input to every downstream system and validate it with deterministic code rather than another model; require human approval for anything irreversible; segregate and label untrusted content, apply access control in the retrieval layer, and keep no secrets in system prompts; and contain failures with sandboxing, rate limits, hard cost ceilings, and full action logging. Filters, refusal instructions, and model-based validation are useful partial measures, not boundaries.

The idea to hold onto is that you cannot make the model trustworthy, so security has to come from the architecture around it: what it is permitted to reach, what happens to what it produces, and which actions require a person, all enforced in code the model cannot talk its way past.

Common questions

How do you secure an LLM application? By enforcing controls outside the model, since the model cannot reliably distinguish instructions from data and can be influenced by any text it receives. The highest-value measures, in order: restrict the tools and credentials the model can access to the minimum required, enforced by your infrastructure rather than described in the prompt; treat everything the model outputs as untrusted input and validate it with deterministic code before it reaches another system; require human approval for irreversible actions such as sending messages, deleting data, or moving money; segregate untrusted content and apply access control to retrieval; and contain failures with sandboxing, rate limits, cost ceilings, and audit logging.

Can prompt injection be prevented? Not fully. It exploits the fundamental design of language models, which process instructions and data in the same channel with no reliable way to tell them apart, so an attacker who can get text in front of the model can influence its behaviour. Filters catch known patterns and miss novel ones, and instructions to resist manipulation are advisory. The realistic approach is defense in depth aimed at limiting impact rather than preventing the attack: least privilege on tools, separation between consuming untrusted content and taking consequential action, validation of output, and human approval for anything irreversible.

What is the OWASP Top 10 for LLM applications? It is the industry reference list of the most significant security risks for applications built on large language models, distinct from the traditional web application list because the attack surface is different. Its categories include prompt injection, sensitive information disclosure, supply chain risks, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Each has associated mitigations, and the recurring themes are least privilege, human oversight for high-impact operations, segregation of untrusted content, and validating model output rather than trusting it.

Why is least privilege so important for AI agents? Because it is the only control that caps the damage from every other failure simultaneously. If a model can be influenced by text it reads, and it holds broad permissions, then influencing it grants access to everything those permissions cover. Narrowing the tool set to what the current task requires, scoping credentials tightly, keeping tokens short-lived, and separating read access from write access all reduce what any successful manipulation can accomplish. The key detail is that the restriction must be enforced by infrastructure, through an explicit allowlist in code, rather than by telling the model which tools it should avoid using.

Should model output be trusted? No. Treat it exactly as you would treat a string submitted by an anonymous internet user, because functionally that is its trust level. If model output becomes a database query, parameterise it; if it becomes part of a web page, encode it; if it becomes a command or file path, validate against a strict allowlist; if it becomes a tool argument, check types and ranges before execution. This category of vulnerability is easy to overlook because the output appears to originate from your own system rather than from an external party, but the model may have been influenced by content that did.

Can I use another AI model to check the first one's output? For some purposes, but not where correctness matters. A second model is subject to the same manipulation as the first, so a validator built from the same material can inherit the weakness it is meant to catch, and its judgments are probabilistic rather than guaranteed. Use deterministic code for anything you need certainty about, such as schema conformance, permission checks, argument validation, and policy rules that can be expressed precisely. Reserve model-based checking for judgments that cannot be coded, like tone or topical relevance, and treat its verdicts as signals rather than as controls.

When should a human approve an AI action? Whenever the action cannot be undone. Sending external communications, deleting or overwriting data, financial transactions, publishing content, changing permissions, and any third-party call with write access all belong behind an approval gate. Read-only operations such as searching, retrieving, drafting, and summarising generally do not. This gate is valuable precisely because it does not depend on the model behaving correctly, converting a class of unrecoverable failures into recoverable ones. It reduces autonomy deliberately, and the right level of autonomy should be chosen based on how bad the worst outcome is rather than assumed.