Tokens and Context Windows Explained

Two terms explain a lot about how AI tools behave and what they cost.

Models don't read words exactly as we do. They split text into tokens: chunks that might be a whole word, part of a word or punctuation.

• Roughly, one token is about three-quarters of an English word (this varies by language and tool).
• Other languages, code and unusual text can use more tokens per word.
• Images, audio and video also get converted into tokens in some systems.

• Pricing: many APIs charge per token for input (what you send) and output (what the model writes).
• Speed: longer outputs take longer.
• Limits: every model has maximum input and output sizes.

The context window is the amount of text the model can consider at once: your instructions, the conversation so far, any documents you provided, and its own reply.

• A larger window lets you include bigger documents and longer chats.
• When a conversation exceeds the window, older content is dropped or summarised. The model can forget earlier instructions.
• Even within the window, models may pay less attention to material buried in the middle of very long inputs.

1. Be concise. Cut irrelevant text.
2. Put key instructions at the start and restate them for long tasks.
3. Start a fresh chat for a new topic.
4. Summarise long conversations and carry the summary forward.
5. For big document sets, use retrieval (RAG) instead of pasting everything.
6. Estimate cost before running large batches.

• A bigger context window doesn't mean better reasoning.
• "Memory" features are separate from the window. They store notes the tool re-inserts into your chats.
• Output limits are often smaller than input limits.

Tokens are the units, the context window is the working memory. Keep both in mind, and you'll get better results at lower cost.