A crawler is software that visits web pages automatically and follows links from one page to the next. Search engines use crawlers to discover and index content, but many other systems do the same thing, including archive tools, monitoring bots, and data collectors.
In cyber security, crawlers matter because they can create heavy load even when they are not malicious. A burst of automated requests can consume bandwidth, slow page responses, and strain small sites that were built for human traffic. Defenders often manage crawler activity with rate limiting, caching, bot filtering, and request monitoring. In attacks, the pattern may look like ordinary browsing at first, but the volume and speed reveal machine behavior that can disrupt availability or enable large-scale content collection.