A token is a unit selected by a model's tokenizer. It might be a word, part of a word, punctuation, whitespace or something less intuitive. Models process token IDs, not English words as such.
A tokenizer converts text to vocabulary IDs. Pieces can include spaces, punctuation or parts of words. Those IDs select learned embedding vectors.
A token is not necessarily a word
When you type a sentence into a language model, a tokenizer transforms its characters into a sequence of IDs. The model uses those IDs as indices into its learned embedding table. Tokenization is one reason the same-looking sentence can cost a different number of tokens across models.
Consider 'unbelievable'. Some tokenizers may keep it together; others may represent pieces such as 'un', 'believ', and 'able'. The actual pieces depend on the particular tokenizer and its vocabulary, so a generic word splitter cannot tell you the exact number of GPT-2 tokens.
What is byte-level BPE?
The GPT-2 family uses a byte-level version of byte pair encoding (BPE). Training starts with smaller units and repeatedly merges common adjacent sequences; the resulting vocabulary can represent frequent patterns with fewer tokens while retaining a way to encode arbitrary input.
Spaces matter. In GPT-2, a leading space can be part of the following token. The same visible word with and without its leading space may map to different IDs. A token ID is simply an address in a fixed vocabulary, not a measure of how important or meaningful a word is.
Why token boundaries affect generation
An autoregressive language model predicts one next token at a time, not necessarily one full word. A continuation might begin with a punctuation mark, a word fragment or an entire frequent word. The decoder then converts the chosen sequence of IDs back into readable text.
Context limits and input costs are usually measured in tokens. A tokenizer designed for one language may divide text in another language differently, which is why token counts should be measured using the exact tokenizer you actually plan to run.
What happens after tokenization?
Each ID selects a row from a learned embedding matrix. Positional information is combined with those vectors, and transformer blocks produce context-dependent representations. The final layer emits a score for every possible next token.
The educational splitter in our basic lab is deliberately illustrative. To see genuine GPT-2 IDs, load DistilGPT-2 in the real model lab. The choice to download and run the model is explicit.
Think about spaces
"cat" compared with " cat"
The visible word is almost identical, but GPT-2 can assign a different token ID when the leading space is included. Run both strings through the real tokenizer to see the actual IDs.
What this does not prove: No exact token IDs are invented here; the demonstration requires running the real tokenizer.
Inspect real GPT-2 token IDs ↗Questions and clear answers
Is a token the same as a word?
No. A token may contain a word, only part of a word, a space or punctuation. It depends on the tokenizer.
Why do token counts vary between models?
Different vocabularies and merge rules split the same input differently; use the target model's tokenizer for an accurate count.
Can a token ID tell me the meaning of a word?
No. The integer is a lookup index. Learned embedding vectors and later contextual computations carry the model's numerical representation.
Research & verificationSubword tokenization
Publication review: Peer-reviewed conference paper published in ACL proceedings.
Independent verification: The homepage visualizer is a toy word/punctuation splitter, not the paper's BPE algorithm. It has unit tests but no third-party audit.
Peer review of a paper does not certify this demo.