Capability benchmarks are repeatable tests that compare AI tools on the same task, using the same inputs and scoring rules. In practice, they measure what a system can actually do, not what a vendor claims it can do. A benchmark might test phishing detection, malware classification, log summarization, incident triage, or code analysis under controlled conditions.
They matter in cyber security because AI tools can affect both defense and risk. Security teams use benchmarks to choose products, spot regressions, and verify that a model performs well on their own workload, not just on a marketing demo. Attackers can also exploit weak evaluation practices by tuning systems to look good on a narrow test while failing on real-world data. Good benchmarks are diverse, reproducible, and resistant to overfitting, so they help teams compare tools, set procurement standards, and validate whether an AI system is reliable enough for operational use.