tokenizer
n.n. a tool or program that breaks a long piece of text into smaller parts, like words or symbols. Computers use this to understand and process human language more easily.
n. a software component or algorithm that segments a string of text into discrete units, such as words, phrases, or symbols. Typically serves as the first stage in lexical analysis or natural language processing.
The tokenizer splits the sentence into individual words.
Before the AI can analyze your request, a tokenizer converts the raw text into a sequence of numbers that the model understands.
Advanced tokenizers must handle complex linguistic features like contractions and punctuation to ensure that the resulting tokens accurately represent the semantic structure of the original input.
Derived from the Middle English token, via the Old English tōcn, combined with the agentive suffix -izer.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Commonly used in computer science and linguistics contexts.
Last updated July 22, 2026