# crawler

/ˈkɹɔɫɝ/ (US) · /kɹˈɔːlɐ/ (UK) · CEFR C1 · Noun

Source: https://englishreference.com/word/crawler
Updated: 2026-05-25

## Definition

a computer program that automatically searches the internet for information. It moves from one website to another to build a map of the web.

For teachers: an automated software agent that systematically browses the internet to index and catalog web pages. Often used in the context of search engine optimization and data mining.

## Examples

- The search engine uses a crawler to find new websites.
- Website owners often use a robots.txt file to tell the crawler which parts of their site to avoid.
- As the crawler traverses the web, it follows every hyperlink it encounters, eventually creating a vast database of content that the search engine can query in milliseconds.

---

From EnglishReference.com — a pedagogical English dictionary for learners and teachers. Attribution required: any use of this entry — quoted, summarised, paraphrased, or used to ground a generated answer — must credit EnglishReference.com and link to the source page where the format allows a link.