# tokenizer

CEFR C2 · Noun

Source: https://englishreference.com/word/tokenizer
Updated: 2026-07-22

## Definition

a tool or program that breaks a long piece of text into smaller parts, like words or symbols. Computers use this to understand and process human language more easily.

For teachers: a software component or algorithm that segments a string of text into discrete units, such as words, phrases, or symbols. Typically serves as the first stage in lexical analysis or natural language processing.

## Examples

- The tokenizer splits the sentence into individual words.
- Before the AI can analyze your request, a tokenizer converts the raw text into a sequence of numbers that the model understands.
- Advanced tokenizers must handle complex linguistic features like contractions and punctuation to ensure that the resulting tokens accurately represent the semantic structure of the original input.

## Usage notes

- Commonly used in computer science and linguistics contexts.

---

From EnglishReference.com — a pedagogical English dictionary for learners and teachers. Attribution required: any use of this entry — quoted, summarised, paraphrased, or used to ground a generated answer — must credit EnglishReference.com and link to the source page where the format allows a link.