The "T" in ChatGPT stands for Transformer. So does the "T" in BERT. Almost every AI tool you have used in the last few years, from Gemini answering questions to Google Translate to the coding assistant in your editor, runs on the same basic design that a small team of researchers published in 2017.
That design solved a problem that had held language AI back for years: older models read text one word at a time, like a student reading with a finger under each word. Transformers read the whole passage at once and work out which words relate to which. That single change made it possible to train models on huge amounts of text using thousands of GPUs in parallel, and the large language models we use today came out of that.
When I teach transformers, I avoid starting with the famous architecture diagram. It scares people. Instead we follow one sentence through the model, step by step, and the diagram makes sense at the end. That is what this page does. You will learn what came before transformers, the 2017 paper, each part of the architecture in plain words, the difference between GPT-style and BERT-style models, and where transformers show up beyond text.
What is a transformer model in AI?
A transformer model in AI is a type of neural network built to handle sequences, such as sentences, code or audio, by looking at every part of the sequence together and learning how the parts relate.
Three ideas define it:
- Parallel reading. It takes in the whole input at once instead of word by word.
- Attention. Every word (more precisely, every token) can look at every other token and decide how much each one matters to its own meaning.
- Stacked layers. The same kind of block repeats many times, and each layer builds a richer understanding than the one before.
The transformer is an architecture, not a single model. Think of it like the design of a building. GPT, Gemini, Claude, Llama, BERT and many others are different buildings made from the same blueprint, with different sizes, training data and tweaks.
Not the toy, not the electrical one
If you searched "transformer model" and saw robots or power stations, you are not alone. The Transformers toy line (Optimus Prime and friends) and the electrical transformer (the box outside your building that steps voltage up or down) share only the name. On this page, a transformer is always the AI architecture.
Transformer model paper: "Attention Is All You Need" (2017)
The transformer model was introduced in the paper "Attention Is All You Need" by Ashish Vaswani and seven co-authors, most of them at Google, posted on arXiv in June 2017 (arXiv 1706.03762) and presented at the NeurIPS conference that year.
The paper had a bold idea in its title. Earlier models used attention as an add-on to recurrent networks. The authors showed you could drop the recurrent part entirely and build a model only from attention and simple feed-forward layers.
What the paper actually did:
- Proposed the encoder-decoder transformer architecture.
- Tested it on machine translation, English to German and English to French.
- Reported better translation quality than earlier systems on those benchmarks, while training faster because the work could be split across hardware in parallel.
The original paper was about translation, not chatbots. Nobody in that paper was talking about ChatGPT. Within about a year, though, OpenAI released the first GPT model and Google released BERT, both built on the transformer, and the architecture spread to nearly every corner of AI.
I recommend every student read at least the abstract and look at Figure 1 of the paper once. It is short, free on arXiv, and it feels good to see the original source of the thing everyone is talking about.
What came before transformers? RNNs and LSTMs
Before transformers, language models mostly used RNNs (recurrent neural networks) and LSTMs (long short-term memory networks), which read text one word at a time and passed a running memory forward.
Picture reading a long WhatsApp forward and only being allowed to keep a single sticky note of everything so far. Each new word, you update the note and throw the old one away. By the end of a long paragraph, details from the start get blurred. RNNs had exactly this problem with long sentences. LSTMs, introduced in the 1990s, added gates that helped them hold on to important information longer, and they powered a lot of translation and speech systems in the 2010s.
Two problems remained:
- Long-range memory was still weak. Linking a word at the end of a long passage to one at the start was hard.
- They were slow to train. Word 10 could not be processed until word 9 was done. You cannot split that work across a thousand GPUs.
Transformers fixed both. Attention lets any token connect directly to any other token, no matter how far apart. And because all tokens are processed at once, training can run in parallel across huge clusters.
| RNN / LSTM | Transformer | |
|---|---|---|
| How it reads text | One token at a time, in order | All tokens at once |
| Long-range connections | Weak, fades over distance | Direct, through attention |
| Training on GPUs | Hard to parallelise | Highly parallel |
| Typical use today | Small or legacy systems | LLMs, translation, vision, speech |
That parallelism is the quiet hero of the story. Bigger data and bigger models only became practical because transformers could use modern hardware efficiently.
Transformer model architecture explained step by step
The transformer architecture turns text into numbers, adds position information, lets tokens share context through attention, refines each token with feed-forward layers, and repeats this across many layers before producing an output.
Let us follow one sentence: "Priya bought chai for her friend."
- Tokenisation. The text is split into tokens, small pieces like "Pri", "ya", " bought", " chai". Each token gets an ID number from a fixed vocabulary.
- Embeddings. Each token ID is turned into a long list of numbers called an embedding. Similar words end up with similar lists, so "chai" sits closer to "coffee" than to "cricket".
- Positional encoding. Because the model reads everything at once, it has no built-in sense of order. Position information is added to each embedding so that "Priya bought chai" and "chai bought Priya" look different. The original paper used sine and cosine patterns for this; newer models use other methods.
- Self-attention. Each token looks at all the others and pulls in what it needs. "her" learns to connect strongly with "Priya". The details are on our attention mechanism page.
- Feed-forward layer. Each token's updated vector then passes through a small neural network on its own, which transforms and enriches it. A lot of the model's stored knowledge is thought to live in these layers.
- Residual connections and normalisation. Around each step, the input is added back to the output and the numbers are rescaled. These tricks keep training stable when you stack many layers.
- Repeat. Steps 4 to 6 form one transformer block. The original base model stacked 6 blocks in the encoder and 6 in the decoder; today's large models stack many more.
- Output. A final layer turns the last vectors into scores for every token in the vocabulary, and the model picks the next token (or a label, depending on the task).
Encoder vs decoder: GPT vs BERT style transformers
An encoder reads and understands the full input, a decoder generates output one token at a time, and different transformer families use one or both halves.
The 2017 model had both: the encoder read the English sentence, and the decoder wrote the German one. Later models found they often needed only one side.
| Type | How it sees text | Good at | Well-known examples |
|---|---|---|---|
| Encoder-only | Looks at words on both sides of each token | Understanding: search, classification, sentiment | BERT and its variants |
| Decoder-only | Looks only at earlier tokens, predicts the next | Generating: chat, writing, code | GPT series, Llama, and most chat LLMs |
| Encoder-decoder | Encoder reads input, decoder writes output | Turning one sequence into another: translation, summarising | Original transformer, T5, Whisper |
Why chatbots are decoder-only
A decoder-only model is trained with one simple game: given the text so far, predict the next token. Because it can only look backwards (a rule called causal masking), it learns to continue text naturally. Scale that up with enough data and you get a model that can answer questions, write essays and generate code, all as "continue this text".
Why BERT is encoder-only
BERT (released by Google in 2018) learned by filling in hidden words in the middle of sentences, looking both left and right. That makes it strong at understanding text and producing useful embeddings, but it is not designed to write long answers. Google has said BERT was used to improve its search results.
A common interview question in India is "what is the difference between BERT and GPT?". If you can say "encoder-only for understanding, decoder-only for generation" and explain masking, you are ahead of most candidates.
How is the transformer model used in LLMs like ChatGPT?
In LLMs like ChatGPT, Gemini and Claude, a very large decoder-style transformer generates replies by predicting one token at a time, adding it to the text, and repeating.
When you type "Write a two-line poem about Mumbai rain", here is what happens inside:
- Your message (plus a hidden system prompt and earlier chat) is tokenised.
- All tokens pass through the transformer's stack of layers together.
- At the top, the model outputs a probability for every possible next token. Maybe "Grey" scores high.
- One token is chosen, appended to the input, and the whole process runs again for the next token.
- This loop continues until the model produces a stop token or hits a length limit.
So a long answer is really hundreds of small predictions, each one made with full attention over everything written so far. That is also why longer chats cost more: every new token has to attend to a bigger and bigger context window.
Modern chat models add a lot on top of the raw transformer: instruction tuning, human feedback training, safety layers, tool use. Model makers do not publish every detail, and designs change with each release. The core engine, though, is still the transformer. For the full picture of how LLMs are built and trained, see what an LLM is.
How is a transformer model trained?
A transformer is trained by showing it huge amounts of data, asking it to predict missing or next tokens, measuring how wrong it was, and adjusting its internal numbers (weights) slightly to be less wrong next time.
Training usually happens in stages:
- Pre-training. The model reads a vast amount of text and learns to predict the next token (GPT style) or fill in blanks (BERT style). Nobody labels this data by hand; the text itself provides the answers. This stage needs large GPU clusters and is the most expensive part.
- Fine-tuning. The pre-trained model is trained further on smaller, focused data, such as question-answer pairs or a company's support tickets. Our fine-tuning page covers this.
- Alignment. Chat models get extra training so their answers follow instructions and stay helpful and safe, often using human or AI feedback.
The mechanics are the same as any neural network: compute a loss, use backpropagation to find which weights caused the error, nudge them with an optimiser, repeat billions of times. What makes transformers special is how well this process scales. In general, more data, more parameters and more compute have produced more capable models, although researchers debate how long that trend continues.
My contrarian view for students: you do not need to pre-train a transformer to understand one. Fine-tuning a small open model on a free GPU notebook teaches you most of the ideas for a tiny fraction of the cost.
Transformer model examples beyond text: vision, speech and more
Transformers now power AI for images, speech, video, code, biology and even game graphics, because almost anything can be cut into tokens.
- Vision. The Vision Transformer (ViT), from Google researchers in 2020, splits an image into small square patches and treats each patch like a word. Many image models now use this idea.
- Speech. OpenAI's Whisper, a speech recognition model, uses an encoder-decoder transformer that turns audio into text. Speech models like this are handy for Indian languages and mixed Hinglish audio, though accuracy varies by language.
- Image generation. Several modern image and video generators use transformer blocks inside their systems, often combined with diffusion techniques.
- Code. Coding assistants are transformer LLMs trained heavily on source code.
- Science. Attention-based models have been applied to protein structure and other scientific data.
- Games. Nvidia announced that DLSS 4 uses a transformer model for upscaling graphics, which is why "DLSS transformer model" shows up in searches. Check Nvidia's own docs for current details.
The pattern is always the same: turn the input into a sequence of tokens, give each one an embedding, and let attention find the relationships.
What are the limitations of transformer models?
The main limitations of transformers are that attention gets expensive as inputs grow, they need enormous data and compute to train well, and they can produce confident but wrong answers.
Cost grows fast with length. In standard attention, every token compares itself with every other token. Double the input length and that comparison work roughly quadruples. That is the reason context windows have limits and long prompts cost more. Engineers use many tricks to soften this, but the pressure is real.
Training is expensive. Building a frontier transformer takes huge GPU clusters and datasets that only a handful of companies can afford. That concentration of power is a real issue, and one reason open models matter.
They predict, they do not verify. A transformer produces the most likely next token, not a checked fact. That is the root of AI hallucination.
They are hard to interpret. We can see attention scores, but explaining why a model with billions of weights produced a specific answer is still an open research problem.
Researchers keep testing alternatives, such as state space models, and hybrids that mix them with attention. As of 2026, transformers remain the dominant design for large language models, but I would not bet my career on any architecture being permanent. Learn the ideas (tokens, embeddings, attention, scaling) and you will understand whatever comes next.
Transformer model example in Python: try one in 3 lines
The quickest way to run a transformer model yourself is the open source Hugging Face Transformers library in Python, which can load a small pre-trained model in a few lines.
Install the library (pip install transformers torch) in Google Colab or on your laptop, then run:
from transformers import pipeline
generator = pipeline("text-generation", model="gpt2")
print(generator("The best street food in Delhi is", max_new_tokens=25))This downloads GPT-2, a small decoder-only transformer OpenAI released in 2019, and asks it to continue your sentence. The output will be rough compared with modern chatbots, which is the point: you are seeing a raw pre-trained transformer without chat training on top.
Next steps that teach a lot:
- Swap the task to "sentiment-analysis" to use an encoder-style model instead.
- Try a translation pipeline to see an encoder-decoder model at work.
- Load a tokenizer and print the tokens for a Hindi sentence versus an English one, and notice how token counts differ.
Library versions and default models change, so if a line errors, check the current Hugging Face documentation. Preparing for interviews on this? Our AI interview prep section has transformer questions with model answers.
Key takeaways
- A transformer is a neural network architecture that reads all tokens at once and uses attention to learn how they relate.
- It was introduced in the 2017 Google paper "Attention Is All You Need" (arXiv 1706.03762), originally for machine translation.
- Transformers replaced RNNs and LSTMs mainly because they train in parallel and connect distant words directly.
- Core parts: tokens, embeddings, positional encoding, self-attention, feed-forward layers, stacked into repeated blocks.
- GPT-style models are decoder-only (generation), BERT-style models are encoder-only (understanding), and some use both.
- Transformers now power vision, speech and code models too, but attention cost grows quickly with input length.