Tokens in AI Explained: How LLMs Count Words, Cost and Memory

What are tokens in AI? (tokens in AI meaning)

Tokens in AI are the basic units of text that a language model processes: whole words, pieces of words, single characters, spaces or punctuation marks.

A large language model cannot read letters the way you do. It only works with numbers. So every AI system that handles text needs a step that converts text into a list of numbers and back again. The pieces it converts are tokens, and the tool that does the cutting is called a tokenizer.

A few quick facts about tokens:

• Common words are often one token. Words like "the", "and" or "India" usually map to a single token in English-focused tokenizers.
• Rare or long words get split. A word like "tokenization" may become two or three tokens.
• Spaces and punctuation count. A leading space is often glued to the next word, and a full stop or comma can be a token of its own.
• Each token has an ID number. The model sees a sequence like 40, 3021, 7754 rather than letters.

Every model family has its own tokenizer with its own vocabulary, typically containing tens of thousands to a few hundred thousand tokens. That is why the same sentence can produce a different token count in ChatGPT, Claude and Gemini.

A quick note to avoid confusion: "tokens" in crypto or in login systems mean something completely different. On this page, a token always means a unit of text for an AI model.

What is a token in AI, with example?

A token is one chunk of your text after a tokenizer has split it, and the easiest way to understand it is to watch one sentence being split.

Take this sentence:

I am learning AI in Bengaluru.

A typical English tokenizer might split it roughly like this (exact splits vary by model):

"I" | " am" | " learning" | " AI" | " in" | " Beng" | "aluru" | "."

That is about 8 tokens for 6 words. Notice three things:

1. Common words like "am", "learning" and "in" stay whole, each carrying its leading space.
2. "Bengaluru" is less common in the training text, so it may be broken into two pieces.
3. The full stop is its own token.

Now each token is swapped for its ID number from the tokenizer's vocabulary, something like a list of eight integers. The model processes those integers, predicts the next token ID, then the next, and the tokenizer converts the output IDs back into text you can read.

If you want to see this yourself, several AI companies offer free online tokenizer tools where you paste text and see the pieces highlighted in colour. I strongly recommend trying one for five minutes. Seeing "Bengaluru" or your own name split into fragments makes the idea stick far better than reading about it.

How does tokenization work in LLMs?

Tokenization in LLMs works by learning a vocabulary of frequent text pieces from a large training corpus, then splitting any new text into the longest pieces that exist in that vocabulary.

The most common method is called Byte Pair Encoding (BPE), adapted for language models from a 2015 research paper by Sennrich and colleagues. Here is the idea in a simple flow:

1. Start small. Begin with a vocabulary of single characters or bytes.
2. Count pairs. Scan a huge amount of text and find which two pieces appear next to each other most often, for example "t" and "h".
3. Merge the winner. Add "th" as a new token in the vocabulary.
4. Repeat many thousands of times. Frequent pairs keep merging into bigger pieces: "th" plus "e" becomes "the", and so on.
5. Stop at a target size. The final vocabulary holds common whole words plus useful word parts like "ing", "tion" or "pre".

When you later type a message, the tokenizer uses those learned merges to split your text. Frequent words become one token; rare words fall back to smaller pieces, and in the worst case to single bytes. That fallback is why a tokenizer can handle any text, including emojis, code and names it has never seen.

Other methods such as WordPiece and SentencePiece follow a similar spirit with different details. You do not need to memorise them. The key insight is that the vocabulary reflects whatever text was common during training, which matters a lot for Indian languages, as you will see below.

Tokens are also what a transformer model operates on. Each token ID is turned into a vector called an embedding, and attention runs across those vectors.

How many tokens is a word? The 4-character rule of thumb

For ordinary English text, one token is roughly 4 characters or about three quarters of a word, so 100 tokens is about 75 words.

That rule of thumb comes from OpenAI's own guidance, and it is close enough for most planning. Here is how it scales:

Keep three warnings in mind:

• It is only an estimate. Every model's tokenizer is different, so the real count can be higher or lower.
• Code, numbers and unusual formatting use more tokens. Long numbers, URLs, tables and code with lots of symbols often split into many small pieces.
• Non-English text often uses more, sometimes much more, for the same meaning.

For exact numbers, use the official token-counting tool or API from the provider you are using. Most major AI providers offer one.

Why do Hindi and Indian languages use more tokens?

Hindi and other Indian languages often use more tokens than English because most tokenizers were built from training data that was mostly English, so their vocabularies contain fewer ready-made pieces for Indian scripts.

Remember how BPE works: frequent text pieces get merged into single tokens. If a tokenizer saw far more English than Hindi, Tamil or Bengali during training, it learned many whole-word tokens for English but fewer for Devanagari, Tamil or Bengali script. A Hindi word may then be split into several small pieces, sometimes down to individual characters or bytes.

What this means for you in practice:

• Higher cost. The same message in Hindi can cost more through an API than its English version, because you pay per token.
• Less room in the context window. A long Hindi document fills a model's limit faster than an English one of similar meaning.
• Sometimes slower replies, since the model has to generate more tokens to say the same thing.
• Hinglish behaves differently again. Hindi written in Roman script ("aap kaise ho") uses the Latin alphabet, so it often tokenizes more efficiently than Devanagari, though still less neatly than standard English.

The good news: this gap has been shrinking. Newer tokenizers from global labs include more multilingual text, and Indian AI teams have built models and tokenizers designed with Indian languages in mind. Exact ratios vary a lot by model and language, so test your own text in the provider's token counter rather than trusting a single number you read online.

My view: if you are building a product for Bharat, not just for English-speaking metros, token efficiency in Indian languages should be one of the first things you test when picking a model.

How do tokens affect AI cost?

AI APIs charge per token, usually quoted as a price per million tokens, with separate rates for input tokens (what you send) and output tokens (what the model writes).

Output tokens are typically priced higher than input tokens, because generating text one token at a time takes more computing work than reading it. Bigger, more capable models cost more per token than smaller ones.

Here is a simple worked example with made-up prices, just to show the maths:

Example rates (illustrative only):
  Input:  100 rupees per 1 million tokens
  Output: 400 rupees per 1 million tokens

One student question:
  Input  = 500 tokens (question + instructions)
  Output = 300 tokens (answer)

Cost per question:
  Input  = 500 / 1,000,000 x 100 = 0.05 rupees
  Output = 300 / 1,000,000 x 400 = 0.12 rupees
  Total  = about 0.17 rupees

10,000 questions a day = about 1,700 rupees a day

Tiny per message, large at scale.

Two more cost details beginners miss:

1. Chat history is re-sent. In a multi-turn chat, earlier messages are usually sent again with every new question, so input tokens grow as the conversation grows.
2. Some models use hidden "reasoning" tokens. Reasoning models may think through steps internally, and providers generally bill those as output tokens even if you do not see them.

Free chatbot apps hide all this from you with daily message limits instead of bills. Real prices change often, so always check the provider's official pricing page.

What is a token limit? Tokens and the context window

A token limit is the maximum number of tokens a model can handle, and it comes in two forms: the context window (total tokens it can consider at once) and the maximum output length (tokens it can write in one reply).

The context window is the model's working memory. Your instructions, the chat history, any uploaded files and the model's own reply all have to fit inside it. If the total goes over the limit, something has to give: the app may refuse the input, cut off older messages or summarise them.

A simple picture:

• Context window: the size of the desk. Everything the model is looking at must fit on it.
• Max output tokens: how many pages the model is allowed to write before it must stop.
• Your prompt plus files: the papers you have already put on the desk.

Signs you have hit a token limit:

1. The app says your message or file is too long.
2. The answer stops mid-sentence.
3. In a very long chat, the model forgets details you shared near the start.

Context windows have grown enormously in the last few years, and many current models advertise hundreds of thousands of tokens, with some going higher. Exact limits differ by model, plan and API, and they change, so check the current documentation.

That last point matters. Models can still miss details buried in the middle of a very long input, so put the most important instructions clearly at the start or the end.

Why do AI agents use so many tokens?

AI agents use many tokens because they work in a loop, and each step usually sends the growing history of instructions, tool results and earlier reasoning back to the model.

A normal chat is one question and one answer. An AI agent is different. Say you ask an agent to "compare three laptops under 60,000 rupees and make a table". It might:

1. Read your request plus a long system prompt describing its tools.
2. Call a search tool and receive several pages of results.
3. Read those results (more input tokens) and decide what to check next.
4. Open product pages, each adding hundreds or thousands of tokens.
5. Write notes to itself, then finally write the table.

Each step re-sends much of what came before, so token use can climb quickly. A task that feels like "one question" to you may consume many times the tokens of a simple chat reply.

Tool descriptions also cost tokens. When an agent is connected to many tools, for example through MCP, the description of every available tool is often included in the context, even before any tool is used.

My contrarian take: for many everyday tasks, a single well-written prompt beats an agent. Agents shine on genuinely multi-step work, but if one good prompt can do the job, it will usually be faster, cheaper and easier to check.

Tokens vs words vs embeddings: what is the difference?

Words are what humans write, tokens are the chunks a model splits text into, and embeddings are the lists of numbers that represent each token's meaning inside the model.

The journey of your text looks like this:

• Words: "Mumbai rains are heavy"
• Tokens: "Mumbai" | " rains" | " are" | " heavy" (in a typical tokenizer)
• Token IDs: one integer per token, looked up in the tokenizer's vocabulary
• Embeddings: each ID is mapped to a long vector of numbers that captures meaning and context

Why the distinction matters:

• Token counts decide cost and limits.
• Embeddings decide how the model "understands" similarity. Words with related meanings end up with vectors that sit close together.

Embeddings also power search features such as RAG, where documents are turned into vectors so the system can find the passages most relevant to your question. To go deeper, read what embeddings are.

I find many beginners mix up "token" and "embedding" because both involve numbers. Here is the shortcut I use in class: a token ID is like a roll number, it just identifies the student. An embedding is like a full report card, it describes the student in detail.

How to use fewer tokens: practical tips for students and builders

You can cut token usage by sending only what the model needs, asking for shorter outputs and choosing the right model for the job.

Practical tips that work:

• Trim your prompt. Remove repeated instructions, long greetings and pasted text the model does not need.
• Ask for a format and length. "Answer in 5 bullet points" or "under 100 words" keeps output tokens in check.
• Start fresh chats for new topics. A long old conversation drags its history into every new question.
• Summarise long documents first. Ask for a summary of a large file, then work from that summary.
• Pick a smaller model for simple jobs. Formatting, classifying or short replies rarely need the biggest model.
• Use retrieval instead of pasting everything. Systems built with RAG send only the most relevant chunks of a document instead of the whole thing.
• Check, do not guess. Use the provider's token counter when cost or limits matter.

A good prompt engineering habit is to write clearly rather than briefly. Cutting important context to save a few tokens usually backfires, because the model then guesses and you spend more tokens fixing the answer.

Understanding tokens is a small skill with a big payoff. It explains AI pricing, long-chat forgetfulness and "message too long" errors in one go, and it is a common question in AI interviews. Practice explaining it in your own words with our AI interview prep questions.

Key takeaways

• Tokens are the chunks of text (words, word pieces, spaces, punctuation) that AI language models read and write.
• For English, one token is roughly 4 characters or about three quarters of a word, so 100 tokens is about 75 words.
• Each model has its own tokenizer, so the same text can have different token counts across ChatGPT, Claude and Gemini.
• Hindi and other Indian languages often need more tokens than English for the same meaning, which raises cost and fills context faster.
• API pricing is per token, with output usually priced higher than input, and context limits are measured in tokens too.
• AI agents use many tokens because each loop step re-sends history, tool descriptions and tool results.