Token
A token is the small chunk of text, roughly a word or part of a word, that a language model reads and generates one piece at a time.
Key takeaways
- A token is the basic text unit a language model reads and generates, roughly three-quarters of a word in English.
- Tokenization splits text into these units before a model can process it, mapping each to a numeric vector.
- AI providers typically charge and rate-limit based on token count, not word or character count.
- A model's context window is measured in tokens, which is what sets its real limit on input and output length.
- Text that doesn't tokenize cleanly, like some non-English languages or code, can require more tokens for the same content.
What is a token in AI?
A token is the basic unit of text a language model processes. Before a model can read a sentence, it breaks that sentence into tokens using a process called tokenization — tokens are often whole short words, but longer or less common words get split into multiple sub-word pieces. As a rough rule of thumb, one token is about four characters of English text, or roughly three-quarters of a word.
Why tokenization matters
Models don't see raw letters or full words; they see a sequence of token IDs, each mapped to a vector the model can do math on. This is also why models handle some languages, code, or unusual formatting less efficiently than plain English — anything that doesn't tokenize cleanly takes more tokens to represent the same amount of information.
Tokens and cost
Most AI providers charge and rate-limit based on the number of tokens processed, both in the prompt sent to the model and the response it generates. That's why long conversations, large documents, or verbose prompts cost more and take longer to process than short ones.
Tokens and the context window
A model's context window is measured in tokens, not words or characters. Understanding tokens is what makes context window limits — like "128,000 tokens" — concretely meaningful, since it caps how much conversation, document, or code a model can consider at once.
Frequently asked
- How many words is one token?
- As a rough estimate, one token is about four characters or roughly three-quarters of a word in English, though this varies by language and content.
- Why do AI companies charge by tokens?
- Tokens directly correspond to the computational work a model does, so pricing by token count reflects the actual processing cost of a request.
- What is the difference between a token and a word?
- A word is a unit of language; a token is a unit a model's tokenizer produces, which can be a whole word, part of a word, punctuation, or even a single character.