Evals are test or evaluation procedures used to measure whether an AI system behaves reliably for a task. They can include benchmark prompts, hidden test sets, safety checks, regression tests, and scenario-based reviews. In AI security, evals matter because they are one of the few ways to catch brittle behavior before a model or tool chain is deployed to users.
In real defenses, teams use evals to verify that prompt changes, model updates, new tools, or configuration edits do not weaken output quality or increase risky behavior. In attacks, weak or missing evals can let unsafe changes slip into production, such as models that leak data, follow malicious instructions, or fail under edge cases. Good evals help define acceptance criteria, reduce release risk, and prove that a system still works after modifications.