What is a jailbreak?
A jailbreak is an input crafted by a user to get a language model to set aside the rules it was trained and instructed to follow. The user is the attacker, and the target is the model’s own behavior. Common techniques include roleplay that casts the model as an unrestricted character, framing a forbidden request as fiction or research, splitting a request into innocent-looking pieces, encoding it in another language or a cipher, and long conversations that gradually shift what the model treats as normal. New techniques appear constantly and vendors patch them in training, so a jailbreak that works on one model version often fails on the next.
A jailbreak is different from prompt injection. In a jailbreak, the person typing the prompt attacks the model’s rules directly. In prompt injection, malicious instructions arrive hidden inside content the model processes on a legitimate user’s behalf, such as an email or a web page.
Why jailbreaks matter
For a company deploying an assistant, a successful jailbreak turns a tool into a liability. A retailer’s shopping assistant that can be talked into disparaging a competitor, inventing a promotion, or producing offensive text creates legal and reputational exposure. Where an assistant has tools, a jailbreak can be the first step toward an unauthorized action. The defense is layered: a clear system prompt, output guardrails that do not depend on the model’s cooperation, strict tool permissions, and AI red teaming before launch to find the techniques that work against your particular setup.