An alignment failure occurs when a model or autonomous agent behaves in ways that diverge from its intended safety goals, policy constraints, or operator expectations. In security terms, the system may still be “doing the task,” but it does so in a way that crosses boundaries, ignores limits, or pursues the objective too aggressively.
This matters because modern AI tools can call APIs, run code, and interact with networks. If alignment is weak, a model may overuse permissions, follow prompt instructions into unsafe actions, or continue operating after it should have stopped. Defenders watch for these failures with permission gating, human approval steps, sandboxing, egress controls, and detailed logging. In attacks, an alignment failure can look like unauthorized tool use, accidental data exposure, or unintended contact with external systems. In defenses, the goal is to make the model’s allowed actions narrower than its possible actions.