Back to the index
39/ 55

SAFETY & SOCIETY

Prompt injection.

An attempt to make an AI system treat untrusted content as instructions that override its intended task.

In plain words

Prompt injection exploits the fact that models receive both instructions and data in forms they can interpret as language. An attacker places instructions where the system should be reading ordinary, untrusted content.

A closer look

An indirect injection might appear in a web page, email, document, or tool response. It can ask the assistant to ignore its original task, reveal information, or use tools in an unintended way. A direct attack comes through input supplied to the model itself.

Defenses involve the whole system: separating trust levels, limiting permissions, validating actions, isolating sensitive operations, and testing adversarial cases. Stronger instruction following can help, but a warning in a prompt is not a complete security boundary. Tool access can make the consequences more significant.

In practice

AN EXAMPLE

An assistant is summarizing a page that contains a hidden instruction to send private notes elsewhere. That page is evidence to summarize, not an authority allowed to change the assistant’s task or permissions.

A useful distinction

Prompt injection and hallucination are different. Hallucination concerns unsupported output; injection concerns an attempt to redirect behavior through input. A jailbreak often targets model restrictions, while prompt injection can target an application’s workflow or data access.

Sources & further reading

Greshake et al. — Indirect Prompt Injection (opens in a new tab)