The Robots Exclusion Protocol is the convention behind robots.txt, a file that tells automated agents which parts of a site they should or should not crawl. It is a coordination mechanism, not an access-control system. Search engines, crawlers, and many AI agents may read it to decide how to behave, but the file does not authenticate a visitor or prove that a bot is who it claims to be.
In cyber security, that distinction matters. Defenders sometimes use robots.txt to reduce accidental crawling of sensitive paths, limit indexing of admin areas, or signal preferred crawl behavior for benign bots. Attackers, however, can ignore it, spoof a user agent, or use it to discover hidden directories and endpoints. Because of that, sensitive resources still need real controls such as authentication, authorization, rate limiting, and logging. Treat the protocol as guidance for automation, not as a security barrier.