Context budgeting
Estimate whether a prompt plus its expected completion fits a model's context window.
AI & ML · Const
Tokenization
Tokenization divides text into tokens—commonly occurring character sequences that models process. Token count affects how much input and generated output can fit within a model’s context length.
How it works
Tokenization splits text into commonly occurring character sequences called tokens. Models process these tokens sequentially, and the tokenizer determines how much input and generated output fits within a model's context length.
Key points
- Both text-generation and embedding models consume text as token chunks.
- A token is a frequent sequence of characters, not necessarily a complete word.
- Token boundaries can split a single word into multiple pieces.
- Spaces commonly belong to the token beginning a word, affecting token boundaries and counts.
- For English text, use four characters or three-quarters of a word as a rough planning estimate.
- The official tokenizer tool can inspect how a particular string is segmented and counted.
Common mistakes
- For text-generation models, input and output share the model's context budget; exceeding the combined maximum prevents the request from fitting.
- Embedding inputs still have a maximum context length, even though embedding models do not generate output tokens.
Example: Count tokens with tiktoken
from tiktoken import get_encoding
encoding = get_encoding("cl100k_base")
text = "ChatGPT is great!"
tokens = encoding.encode(text)
print(tokens)
print(len(tokens))This Python example encodes text with tiktoken and prints both its token IDs and token count.
When to use it
Prompt analysis
Inspect how spaces, words, and punctuation are split before designing prompts.
Embedding validation
Measure input size before sending embedding requests.
Interview questions
QWhat is a token?
Tokens are chunks of commonly occurring character sequences rather than necessarily whole words. For example, a word may split into multiple tokens, while a short common word may remain one token.
Follow-up: How would you estimate token usage before making a request? Use a tokenizer for the target model; the English rule of thumb is about four characters or three-quarters of a word per token.
QHow can token count be estimated?
A practical English estimate is approximately four characters or 0.75 words per token. This is only a rough estimate, so exact budgeting should use the tokenizer.
Follow-up: Why can the estimate be inaccurate? Token boundaries depend on the actual character sequences, including spaces and uncommon text.
QWhy does tokenization matter for API requests?
Token count matters because the input and generated output together must fit the model's maximum context length. Counting tokens helps determine whether a request can fit before sending it.
Follow-up: What changes for embeddings? Since embeddings models do not produce tokens, only the input must remain below their maximum context length.
Related
Tokenizertiktokencontext lengthembeddings models