robots.txt
n.n. a small text file on a website that tells search engines which pages they are allowed to visit. It helps website owners keep certain parts of their site private from the public.
n. a standard text file used by websites to communicate with web crawlers and other automated bots. It specifies which parts of the domain the agent is permitted to scan or index by following the Robots Exclusion Protocol.
The developer updated the robots.txt file to block search engine crawlers.
You should check your robots.txt file to ensure that private administrative pages are not being indexed.
By carefully configuring the directives within the robots.txt file, webmasters can optimize their crawl budget and prevent sensitive directories from appearing in search results.
Formed from the plural noun robots and the filename extension .txt, denoting a plain text file. The term was established in 1994 by Martijn Koster as part of the Robots Exclusion Protocol.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Usually referred to in the singular as a specific filename; functions as a proper noun for the file itself.
Last updated July 22, 2026