Crawling is the automated fetching of web pages by a search engine bot, often called a crawler or spider. The bot follows links or other discovery signals to retrieve page content so the search engine can later decide whether to index it and show it in search results.
In cyber security, crawling matters because a page may be reachable without being intended for public discovery. If a share link, document, or admin page is crawlable, it can become easier to find through search, increasing exposure beyond the original recipient. Defenders use controls such as noindex directives, authentication, and careful link design to limit what crawlers can access. Attackers also use crawling-like tools to enumerate sites, map hidden pages, and locate exposed data.