What is a token in AI?
In one line
Ages 8 to 10: A token is a small piece of a sentence, like a word or part of a word, that an AI reads.
Ages 11 to 13: An AI splits text into tokens from a fixed list, so common words are one token and rare words become several.
Ages 14 to 18: Tokens are the subword units a tokenizer learns from data; the model predicts one token at a time and its context window is measured in tokens.
A token is a small piece of text that an AI language model reads and writes. A token can be a whole word, part of a word or a punctuation mark. The model splits your message into tokens, works with them as numbers, and writes its reply one token at a time.
Last reviewed: 20 October 2026 6 min read
The idea in one picture
Think of building words from a box of letter-and-chunk tiles. Common words like "the" or "dog" have their own tile. A rarer word is built from smaller tiles that already exist, like "low" + "er" for "lower". An AI language model reads and writes with tiles like these. Each tile is a token.
Worked example. Suppose a model has the tokens "low", "new" and "er" in its list, but not the word "lower". When it meets "lower", it can still handle it as "low" + "er". A really unusual word could be broken all the way down into single letters. The exact split depends on the tokenizer, so different AI systems split the same sentence differently.
How it works, step by step
- Before the model is used, a tokenizer learns a fixed list of tokens from a large amount of text, starting from letters and merging pairs that often appear together.
- When you send a message, the tokenizer splits it into tokens from that list.
- Each token is matched to its own number in the list, and the model works with those numbers.
- The model predicts a likely next token, adds it, and predicts again.
- The finished tokens are turned back into text you can read.
A little deeper (ages 11 to 13)
Why not just use whole words? Because new and rare words appear all the time, and a model with only whole words would get stuck on any word it never saw. Pieces solve that: every new word can be built from pieces the model already knows. Common words still tend to be a single token, and rare words take several. This is also why the prompt you write and the reply you get are, to the model, just long rows of tokens.
Going further (ages 14 to 18)
A widely used method, byte-pair encoding, counts which neighboring symbols appear together most often, merges them into a new token, and repeats many thousands of times. Real vocabularies hold tens of thousands of tokens or more. Inside the model, each token becomes a list of numbers, a vector, and the neural network works on those. The amount of text a model can take in at once, its context window, is measured in tokens, not words. One trade-off: when a tokenizer is learned mostly from English text, other languages can get fewer tokens of their own, so the same message may need more tokens.
Common mix-ups
- Mix-up: "A token is a word."Fact: Often it is, but many tokens are parts of words or punctuation.
- Mix-up: "Every AI splits text the same way."Fact: Each tokenizer has its own list, so splits differ between systems.
- Mix-up: "The model reads letters."Fact: It mostly works with token numbers, not individual letters.
- Mix-up: "Tokens are like arcade tokens or coins."Fact: In AI, a token is a piece of text. Some services count these same text pieces to measure how much is used, but a token is still text, not money.
Try it (no device, 5 minutes)
Make a tiny tokenizer. Write these chunks on paper scraps: "play", "ing", "ed", "jump", "un", "help", "ful", "re", "do", plus two scraps for each letter from a to z. Take turns calling out words: "playing", "unhelpful", "redo", "jumps". Build each one from the fewest scraps. Then call out "skateboard". You can build it only letter by letter, with ten scraps. Which neighboring pairs would you glue into one new chunk, like "sk" or "oa"? Each merge of a common pair makes the row of scraps shorter. That is how a tokenizer grows its list.
Quick check
Is a token always a whole word?
Answer: No. It can be a word, part of a word or a punctuation mark.
No. It can be a word, part of a word or a punctuation mark.How does an AI handle a word it has never seen?
Answer: It builds the word from smaller tokens it already knows, down to single letters if needed.
It builds the word from smaller tokens it already knows, down to single letters if needed.(Ages 14 to 18) What is a context window measured in?
Answer: Tokens: it is the number of tokens the model can take in at once.
Tokens: it is the number of tokens the model can take in at once.Questions
A token is a small piece of text that an AI language model reads and writes, such as a word, part of a word or a punctuation mark. The model splits your message into tokens and writes its reply one token at a time.
New and rare words appear all the time, and a model that only knew whole words would get stuck on them. With pieces, any word can be built from tokens the model already knows.
Common words are often a single token, while rare or long words can take several. The exact count depends on the tokenizer, so it differs between AI systems.
A context window is the amount of text a model can take in at once, measured in tokens. Text beyond it is not part of what the model uses for its next prediction.
Tokenization is the step that splits text into tokens from a fixed list. The list is learned beforehand from lots of text by repeatedly merging pieces that often appear together.
Mechna Academy is a city of machines for curious children aged 8 to 18, opening first to ages 8 to 10. In each mission, a child guesses how a machine decides, then opens its Peek Hatch to check, changes one thing and tests it on a case it has never seen. Children learn how AI works this way, and they never chat with an AI.
For grown-ups: join the waitlistPlaces are limited. The waitlist is for parents and guardians 18 and older.