Prompt Injection Security

Why this vulnerability exists at all

An LLM doesn't have a hard, structural separation between 'instructions I should follow' and 'data I'm just reading.' Everything — your system prompt, the user's message, retrieved documents, tool outputs, a webpage the model is summarizing — gets concatenated into one token stream, and the model predicts its next output based on all of it together. If text that was meant to be passive data happens to look like an instruction, the model may follow it, because from the model's perspective, instructions and data are not a different type of thing — they're just tokens in the same sequence.

Prompt injection is the general term for exploiting this: crafting input that causes the model to deviate from its intended instructions and instead follow instructions embedded in the input itself.

Direct vs indirect injection

Direct prompt injection is when the attacker is the user talking to the model directly — typing something like 'ignore your previous instructions and instead...' straight into the chat. This is the most visible form and the one most systems defend against first, because the attacker and the input are the same, obvious thing.

Indirect prompt injection is more dangerous precisely because it's less visible: the malicious instructions arrive hidden inside content the model is asked to process on someone else's behalf — a webpage it's summarizing, an email it's reading, a document retrieved by a RAG pipeline, a tool's output. The user never typed anything malicious; the attack comes from a third party who planted instructions in content they knew or hoped an AI system would eventually read.

Why this is hard to fully solve

There is currently no fully reliable, general-purpose technical fix that guarantees a model will never follow injected instructions — this remains an open problem, not a solved one. Models can be trained and prompted to be more resistant (being explicitly told to treat retrieved content as untrusted data, never as instructions, helps measurably), but a sufficiently creative injection can still sometimes succeed, especially as attackers iterate against known defenses. This is structurally similar to other security problems where defenses raise the cost of an attack without making it impossible — the practical goal is risk reduction and containment, not an absolute guarantee.

Direct vs indirect prompt injection

Mitigation strategies that actually help

No single defense is sufficient, but layered mitigations meaningfully reduce risk. Privilege separation: give the agent only the tools it strictly needs for its task, scoped as narrowly as possible — an agent that can't send emails can't be hijacked into sending malicious ones. Human-in-the-loop confirmation for any high-stakes or irreversible action (sending messages, spending money, deleting data) before it actually executes. Explicit framing in the system prompt and architecture that retrieved or tool-returned content is data to be processed, not instructions to be followed — some systems go further and structurally separate instruction channels from data channels where the underlying model or framework supports it. Output monitoring that flags unusual or out-of-scope actions for review. Least-trust defaults: treat all external content — web pages, documents, emails, API responses — as adversarial until proven otherwise, especially in any pipeline with tool-calling agents attached downstream.

What's next

Prompt injection is one specific class of vulnerability. AI red-teaming is the broader practice of systematically probing a model or system for any kind of harmful, unsafe, or unintended behavior before it ships — including but not limited to injection attacks. The next lesson covers how red-teaming works and why it's now a standard part of pre-launch practice.