Top news
Investing in the public benefit OfficialAnthropic NewsOct 18Popular Musk Pledges SpaceX Shares to ‘Every Kid in America’ WebBloomberg TechnologyOct 19Popular Websites are closing their doors to AI agents WebThe Verge AIOct 17 This is the new ChatGPT Voice, powered by GPT Live OfficialOpenAI on YouTubeOct 18 How AI giants will reshape global philanthropy WebAxios AIOct 19 Gemini 4 Argon Really Surprised Me WebTwo Minute PapersOct 19 Google Gemini 4 agents just beat Minecraft with no human intervention WebEngadget AIOct 19 Who could it be? 🧐 OfficialGoogle on YouTubeOct 18 Anthropic gives outside AI watchdog “employee-like” access WebArs Technica AIOct 18 What are self-replicating ‘AI worms’, and why is Anthropic worried about them? WebThe Guardian AIOct 18
GlossaryModelsNew to AI

Token

Also written: Token burn, Tokenization, Token cost, Output tokens, Token limits, Context length, Tokens per query, Per token

Tokens are units of text, like words or parts of words, that AI models process to understand and generate language.

In plain terms

Think of tokens as the building blocks AI uses to read and write. When you give an AI text, it breaks it down into these pieces. These pieces can be whole words, like 'cat' or 'run', or they can be parts of words, like 'ing' in 'running' or 'un' in 'unhappy'.

AI models process these tokens to understand the meaning of your input and to generate their responses. The number of tokens an AI can handle at once is called its context window. For example, OpenAI's GPT-4o Mini has a 2000-token limit for its text-to-speech function, meaning it can process up to 2000 tokens of text at a time.

The expert version

Tokens are discrete units of text that form the input and output of large language models (LLMs). They are typically derived from raw text through a process called tokenization, which can segment text into words, sub-word units (like '##ing' in some tokenizers), or even individual characters. The choice of tokenization strategy impacts vocabulary size, model efficiency, and how well the model handles different languages and rare words. Models have a finite context window, measured in tokens, which limits the amount of information they can consider simultaneously.

Why it matters now

Recent news highlights models with significantly expanded context lengths, such as Google's Gemini 4 Argon reaching one million tokens. This increase allows for more complex reasoning and handling of massive amounts of text in a single interaction. Companies are also exploring ways to manage the costs associated with processing these larger token counts.

Where it shows up 130 items, newest first
How each company uses it
Anthropic Claude
Anthropic is developing systems like TokenEconomizer to manage AI coding costs, implying a focus on token usage and efficiency.
Google Gemini
Google's Gemini 4 Argon model is prominently featured for its massive one million token context window, enabling advanced AI tasks and complex reasoning.
OpenAI ChatGPT
OpenAI's models are being integrated into AI agents that can handle more complex tasks beyond just code, suggesting an increased capacity for processing information.
Related terms

Context WindowTokenizationLLMPrompt EngineeringNatural Language ProcessingEmbeddings

Often confused with
Word
While tokens often correspond to words, they can also be parts of words or
Character
Characters are the smallest units of text, whereas tokens are typical
Origin

The concept of tokens as discrete units for text processing in computational linguistics and natural language processing has been around for decades. In the context of modern LLMs, tokenization became a critical component for handling vast amounts of text efficiently. Source ↗

Try this about 5 minutes

Open ChatGPT or Google Gemini. Type a sentence like 'The quick brown fox jumps over the lazy dog.' Then, try to rephrase it in a few different ways. Notice how the AI understands your request and generates a new sentence, processing each word and punctuation mark as tokens.

Try it: Playbook entries that use thisPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Questions people ask

How many tokens are in a word?
It varies. Common words might be one token, while longer or rarer words might be split into multiple tokens. Punctuation also counts as tokens.
Why do AI models use tokens instead of just words?
Using sub-word tokens allows models to handle a wider range of words, including misspellings and new words, more efficiently than if they had to learn every single word as a unique unit.

First seen on Subvolts Aug 22, 2023. Updated Oct 19, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.