Word
crawler
US/ˈkɹɔɫɝ/
UK/kɹˈɔːlɐ/
Countable noun
Definition
n. a computer program that automatically searches the internet for information. It moves from one website to another to build a map of the web.
n. an automated software agent that systematically browses the internet to index and catalog web pages. Often used in the context of search engine optimization and data mining.
Examples
“The search engine uses a crawler to find new websites.”
“Website owners often use a robots.txt file to tell the crawler which parts of their site to avoid.”
“As the crawler traverses the web, it follows every hyperlink it encounters, eventually creating a vast database of content that the search engine can query in milliseconds.”
Examples
simple
“The search engine uses a crawler to find new websites.”
contextual
“Website owners often use a robots.txt file to tell the crawler which parts of their site to avoid.”
complex
“As the crawler traverses the web, it follows every hyperlink it encounters, eventually creating a vast database of content that the search engine can query in milliseconds.”
Real-World Examples
“The ladybug begins life as a spiky black crawler; the garden tiger moth starts out life as an extravagantly furred caterpillar.” “The moment the number of pages outstripped the number of people who could reasonably classify them, the future belonged to crawlers and ranking systems.” “He noted that while it blocks other unlicensed AI crawlers, it can’t block Google’s since it’s the same one used for Google Search.” Etymology
From crawl (“to move slowly, by dragging the body along the ground”) + -er.
From crawl (“to act in a servile manner”) + -er. From the Australian convict period (1788–1850); a prisoner who was purposely and extensively abused by an overseer (also a convict) and thereby driven to escape but who, finding it impossible to survive in the Australian bush, surrenders to this overseer, who would then have his penal term reduced. The particular crawler was picked for his weak personality and might escape and return a number of times increasing his own penal term each time. According to James Tucker, some convict overseers had their sentences extensively reduced using this odious practice.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Word family
VERB
NOUN
Idioms
01. curb crawler
Domains
- Exam level
- CambridgeCambridgeCambridge English QualificationsThe exam suite from Cambridge University Press & Assessment, from A1 Movers up to C2 Proficiency. The benchmark names double as CEFR labels.C1 AdvancedIELTSIELTSInternational English Language Testing SystemThe 0–9 band test co-owned by the British Council, IDP, and Cambridge. Used for university entry, work, and migration.7.0–8.0TOEFL iBTTOEFL iBTTest of English as a Foreign Language, Internet-Based TestETS's academic English test, the standard route for US university admissions. Scored 0–120, with a 1–6 scale since 2026.1–6 scale: 5.0–5.595–113TOEICTOEICTest of English for International CommunicationETS's workplace English test, big in Japan and Korea. Listening & Reading scores 10–990 and tops out at C1.945–990
- Frequency
- Occasional
- Level
- C1 · Advanced
- Updated
- アルファベット
- 読み方の一覧 →
QR code
englishreference.com/q/crawler