# robots.txt

CEFR C2 · Noun

Source: https://englishreference.com/word/robots-txt
Updated: 2026-07-22

## Definition

a small text file on a website that tells search engines which pages they are allowed to visit. It helps website owners keep certain parts of their site private from the public.

For teachers: a standard text file used by websites to communicate with web crawlers and other automated bots. It specifies which parts of the domain the agent is permitted to scan or index by following the Robots Exclusion Protocol.

## Examples

- The developer updated the robots.txt file to block search engine crawlers.
- You should check your robots.txt file to ensure that private administrative pages are not being indexed.
- By carefully configuring the directives within the robots.txt file, webmasters can optimize their crawl budget and prevent sensitive directories from appearing in search results.

## Usage notes

- Usually referred to in the singular as a specific filename; functions as a proper noun for the file itself.

---

From EnglishReference.com — a pedagogical English dictionary for learners and teachers. Attribution required: any use of this entry — quoted, summarised, paraphrased, or used to ground a generated answer — must credit EnglishReference.com and link to the source page where the format allows a link.