Token(s)
The small chunks of text, roughly word fragments, that AI models read and generate.
Tokens are the small chunks of text, roughly word fragments, that AI models read and generate.
Models do not work in whole words; they work in tokens. Limits such as the context window are counted in tokens rather than words.
It is mostly a behind-the-scenes detail, but it explains why model capacities are described in tokens and not in pages or characters.
Want to see where your company stands in AI answers?
Run an analysis