Attention Mechanism in LLMs and Transformers, Explained With Examples

What is attention in LLM?

Attention in an LLM is a mechanism that lets each token pull in information from the other tokens that matter most to it, with the importance of each one learned from data rather than hard-coded.

Every word in your prompt is first split into tokens and turned into a list of numbers, an embedding. At that point, the embedding for "it" is the same whether the sentence is about a tired animal or a wide road. Attention fixes that. It lets "it" look across the sentence, decide that "animal" (or "road") is the important neighbour, and update its own numbers to carry that meaning.

In plain terms, attention does three things for every token:

• Scores every other token for relevance.
• Turns those scores into weights that add up to 1, like percentages.
• Blends information from the other tokens using those weights.

After attention, each token's vector is no longer just "the word itself". It is the word in context. "Bank" in "river bank" and "bank" in "SBI bank account" end up with different vectors.

If you came here from a quiz: the algorithm LLMs use to focus on specific parts of the input text is the attention mechanism, usually in its self-attention form.

Attention mechanism in deep learning: where did it come from?

The attention mechanism entered deep learning in 2014 through machine translation research, and the 2017 transformer paper made it the main building block of language models.

Before attention, translation models used an encoder that squeezed an entire input sentence into one fixed-size vector, and a decoder that had to write the translation from that single summary. Short sentences worked. Long ones fell apart, because one vector cannot hold every detail of a 40-word sentence.

In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed a fix in their paper "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv 1409.0473). Their idea: while writing each output word, let the decoder look back at all the input words and focus on the most relevant ones. When translating a sentence into Hindi, the model could focus on the matching English word at each step instead of relying on a blurry summary. That focusing step is what we now call attention.

For three years, attention was an add-on to recurrent networks (RNNs and LSTMs). Then in 2017, Vaswani and colleagues at Google published "Attention Is All You Need", which threw away the recurrent part and built a model from attention alone, plus simple feed-forward layers. That model was the transformer.

So the history in one line: attention started as a helper for translation, then became the whole engine.

What is self-attention in LLM?

Self-attention is attention where a sequence looks at itself: every token in the input scores and pulls information from every other token in the same input.

The 2014 version of attention connected two different sequences (the output being written and the input being read). Self-attention, the version at the heart of transformers, works inside one sequence. In the sentence "Ravi told Arjun that he had cleared the GATE exam", self-attention lets "he" check both "Ravi" and "Arjun", and lets "cleared" connect to "exam".

Self-attention vs cross-attention

• Self-attention: queries, keys and values all come from the same sequence. Used in every layer of LLMs.
• Cross-attention: queries come from one sequence and keys and values from another. Used in encoder-decoder models, where the text being generated looks at the input being translated or summarised, and in many image generators, where image features look at the text prompt.

Masked (causal) self-attention

Chat LLMs like the GPT series generate text left to right, so during training a token must not peek at words that come after it. A causal mask blocks those future positions, letting each token attend only to itself and earlier tokens. That is why an LLM's prediction for the next word depends on everything before it, and nothing after.

Students often ask whether self-attention means the model "pays attention to itself". It does not. It means the input pays attention to other parts of the same input.

Query, key and value in attention: a simple library analogy

Query, key and value are three different vectors made from each token: the query says what the token is looking for, the key says what each token offers, and the value is the information that actually gets passed along.

Think of a college library.

• You walk in with a query: "I need a book on Indian economic history for my assignment."
• Every book has a key: the title and subject label on its spine.
• Every book also has a value: the actual content inside.

You compare your query with each key. A book labelled "Economic History of India" matches strongly; "Organic Chemistry" barely matches. Then you read mostly from the strong matches, a little from the medium ones, and almost nothing from the rest. Your notes are a weighted blend of the values.

Attention does exactly this for every token at once. Each token's embedding is multiplied by three learned weight matrices to produce its query, key and value vectors. Then:

1. The query of the current token is compared with the keys of all tokens.
2. Good matches get high scores; poor matches get low scores.
3. The values are blended according to those scores.

The clever part is that the model learns what to put in queries, keys and values during training. Nobody tells it that pronouns should look for nouns. It discovers patterns like that because they help it predict text better.

Attention mechanism formula, explained term by term

The standard attention formula from the 2017 transformer paper is softmax(Q x K^T / sqrt(d_k)) x V, which scores queries against keys, scales the scores, turns them into weights and uses them to mix the values.

Written out in full:

Attention(Q, K, V) = softmax( (Q x K^T) / sqrt(d_k) ) x V

Here is each piece in plain words:

• Q (queries): a table with one row per token, each row being that token's query vector.
• K (keys): the same shape, with each token's key vector.
• K^T: K transposed (rows turned into columns), so the multiplication lines up.
• Q x K^T: multiplies every query with every key, giving a grid of raw relevance scores. With 10 tokens, that is a 10 by 10 grid.
• d_k: the length of each key vector. In the original paper's base model it was 64.
• Divide by sqrt(d_k): the scaling step. With long vectors, dot products can become very large, which pushes softmax into a zone where it puts nearly all weight on one token and learning slows down. Dividing by the square root keeps scores in a healthy range. That is why the paper calls it scaled dot-product attention.
• softmax: turns each row of scores into positive weights that add up to 1.
• x V: uses those weights to take a weighted average of the value vectors.

A tiny numerical example

Say the token "it" scores 2.0 against "animal", 1.0 against "road" and 0.0 against "the" after scaling. Softmax turns these into roughly 0.67, 0.24 and 0.09. So the new vector for "it" is about 67 percent "animal" information, 24 percent "road" and 9 percent "the". Real models do this with thousands of dimensions, but the arithmetic is the same.

How does attention work in transformers, step by step?

Inside a transformer, attention runs in every layer: tokens are projected into queries, keys and values, scored against each other, weighted with softmax, blended, and then passed on to a feed-forward network.

Here is the full flow for one attention layer:

1. Start with token vectors. Each token arrives as a vector, either from the embedding layer plus position information, or from the previous layer.
2. Project. Multiply each vector by three learned matrices to get its query, key and value.
3. Score. Compare each query with all keys (the Q x K^T step).
4. Scale and mask. Divide by sqrt(d_k). In a chat model, apply the causal mask so tokens cannot see the future.
5. Normalise. Apply softmax so each token's scores become weights adding up to 1.
6. Blend. Multiply the weights by the values to get a new context-aware vector for each token.
7. Combine heads. If there are several attention heads (next section), join their outputs and project them back to the original size.
8. Add and pass on. Add the result back to the original vector (a residual connection), normalise, and send it through the feed-forward layer.

Stack dozens of these layers and something interesting happens. Researchers studying trained models have found that early layers often capture simple patterns, like nearby words and grammar, while later layers tend to capture broader meaning. The model is not told to organise itself that way; it falls out of training.

For how these layers fit into the full model, including tokens, embeddings and the encoder versus decoder split, see our transformer model page.

What is multi-head attention?

Multi-head attention runs several attention operations in parallel, each with its own query, key and value weights, so the model can track different kinds of relationships at the same time.

One attention head produces one set of weights per token. But language has many relationships happening together. In "Meera, who lives in Chennai, gave her old laptop to her cousin", the word "gave" relates to who gave (Meera), what was given (laptop) and to whom (cousin). One head trying to capture all of that would blur it.

So transformers split the work:

• The original transformer base model used 8 heads, each working with smaller 64-number vectors instead of one head with 512.
• Each head learns its own projections, so each can specialise.
• The outputs of all heads are joined together and mixed by one more learned matrix.

Researchers inspecting trained models have found heads that appear to focus on things like the previous word, matching brackets or linking pronouns to nouns. Many heads, though, do not map to any neat human-readable job.

Newer LLMs often use variations designed to save memory during generation, such as multi-query attention and grouped-query attention, where several query heads share the same keys and values. The exact designs differ between models and are not always published, so check each model's technical report if you need specifics.

Types of attention mechanism in NLP and LLMs

The main types of attention you will meet are self-attention, cross-attention, masked attention, multi-head attention and efficiency-focused variants that reduce memory or compute.

You may also hear about FlashAttention. It is not a new kind of attention but a faster way to compute standard attention on GPUs, by being smarter about memory. Same math, less waiting.

For exams and interviews in India, the first six rows are the ones you will be asked about. If you can explain scaled dot-product attention and why decoders need a mask, you have covered most questions.

Why does attention get expensive with long context?

Attention gets expensive with long inputs because every token compares itself with every other token, so the work grows roughly with the square of the input length.

With 1,000 tokens, the score grid has 1,000 x 1,000 = 1 million entries per head per layer. With 100,000 tokens, it has 10 billion. Doubling the input length roughly quadruples this part of the work. Engineers call it quadratic cost.

This one fact explains a lot of things beginners find confusing:

• Why context windows have limits. Memory and compute climb steeply as inputs grow.
• Why long chats get slower and pricier. Each new token has to attend to everything before it.
• Why models sometimes miss details in the middle of long documents. Attention is spread across a huge number of tokens, and studies have reported that models often recall information at the start and end of long inputs better than the middle.
• Why RAG still matters. Retrieving the five most relevant paragraphs is often cheaper and more accurate than pasting a 300-page PDF. See what RAG is.

During generation, models store the keys and values of earlier tokens in a KV cache so they do not recompute them every step. That saves time but uses a lot of GPU memory, which is exactly what grouped-query attention tries to reduce.

My contrarian take: a bigger context window is not automatically better. I see students paste entire textbooks into a chat and wonder why the answer is vague. Fewer, more relevant tokens usually beat more tokens.

Does attention mean the model understands?

No, attention is a learned weighting method, not proof of understanding, and attention scores do not fully explain why a model gave a particular answer.

It is tempting to look at an attention map, see "it" pointing strongly at "animal", and conclude the model reasoned like a human. Sometimes the picture is that tidy. Often it is not. A model has many layers and many heads, information mixes through feed-forward layers and residual connections, and researchers have debated for years whether attention weights alone are a faithful explanation of model behaviour.

What attention does give you:

• A powerful, flexible way for tokens to share information.
• A mechanism that scales well to huge datasets and hardware.
• A useful, if partial, window into what a model is focusing on.

What it does not give you:

• Guaranteed facts. A model can attend perfectly and still produce a hallucination.
• A full explanation of a model's decision.

I still think attention is the best single concept to learn if you want to understand modern AI deeply. Once it clicks, transformers, context windows, KV caches and even the cost of your API bill all start to make sense. Practise explaining it out loud with the library analogy; our AI interview prep section has attention questions to test yourself.

Key takeaways

• Attention lets each token weigh every other token in the input and blend in information from the most relevant ones.
• Each token gets a query (what it looks for), a key (what it offers) and a value (the information passed along).
• The core formula is softmax(Q x K^T / sqrt(d_k)) x V, from the 2017 paper "Attention Is All You Need".
• Multi-head attention runs several attention operations in parallel so the model can track different relationships at once.
• Chat LLMs use masked self-attention, so each token sees only earlier tokens.
• Attention cost grows roughly with the square of input length, which is why context windows have limits.