crawler
n. countablen. a computer program that automatically searches the internet for information. It moves from one website to another to build a map of the web.
n. an automated software agent that systematically browses the internet to index and catalog web pages. Often used in the context of search engine optimization and data mining.
The search engine uses a crawler to find new websites.
Website owners often use a robots.txt file to tell the crawler which parts of their site to avoid.
As the crawler traverses the web, it follows every hyperlink it encounters, eventually creating a vast database of content that the search engine can query in milliseconds.
From crawl (“to move slowly, by dragging the body along the ground”) + -er.
From crawl (“to act in a servile manner”) + -er. From the Australian convict period (1788–1850); a prisoner who was purposely and extensively abused by an overseer (also a convict) and thereby driven to escape but who, finding it impossible to survive in the Australian bush, surrenders to this overseer, who would then have his penal term reduced. The particular crawler was picked for his weak personality and might escape and return a number of times increasing his own penal term each time. According to James Tucker, some convict overseers had their sentences extensively reduced using this odious practice.