Tokenizer Trap
A small LLM puzzle about why models can write beautifully and still stumble on character counting.
LLMs read chunks of text called tokens. That makes some simple-looking counting tasks harder.
Puzzle
Try it before the answer.
An LLM can write a coherent paragraph about banana farming, but may answer the number of letters in 'banana' incorrectly unless pushed step by step. Why can a system that writes well fail a tiny counting task?
Hints
- The model does not naturally see text as individual letters.
- Its input is split into tokens.
- A tool or explicit character-level reasoning changes the task.
Solution
Most LLMs operate over tokens, not raw characters. A word may be represented as one token or several token pieces. Character counting is therefore not the native operation. Reliable systems use tools, constrained decoding, or explicit intermediate representations for exact symbolic tasks.
Token View
Compare the word as letters with the pieces shown by a tokenizer. Token boundaries need not match letter boundaries; the tokenizer link below lets you inspect the difference.
Sources & further reading
External resources can change. Check the linked source for current requirements and details.