AI Lab / AI Mechanics
Tokenizer Trap
A small LLM puzzle about why models can write beautifully and still stumble on character counting.
Puzzle
Challenge first.
An LLM can write a coherent paragraph about banana farming, but may answer the number of letters in 'banana' incorrectly unless pushed step by step. Why can a system that writes well fail a tiny counting task?
Hints
- The model does not naturally see text as individual letters.
- Its input is split into tokens.
- A tool or explicit character-level reasoning changes the task.
Solution
Most LLMs operate over tokens, not raw characters. A word may be represented as one token or several token pieces. Character counting is therefore not the native operation. Robust systems use tools, constrained decoding, or explicit intermediate representations for exact symbolic tasks.