Black box AI is an AI system whose internal decision process is not fully visible or easily interpretable. In security work, this matters because operators may know what the model outputs, but not exactly why it made a choice. That makes it harder to detect hidden bias, data leakage, unsafe automation, or model drift before those issues affect access decisions, threat scoring, fraud filters, or incident triage.
Attackers can abuse black box systems by probing them until they reveal patterns, or by exploiting gaps between observed behavior and understood logic. Defenders respond with controls that do not depend on full interpretability: strong logging, bounded use cases, human review for high-impact actions, red-team testing, and continuous monitoring for unexpected output changes. In practice, a secure AI deployment treats the model as one component in a governed system, not as an unquestioned decision-maker.