What is chain of thought prompting in AI?
Chain of thought prompting in AI means asking a language model to reason through a problem step by step, in writing, before it states the final answer.
Remember your school maths teacher saying "show your working"? Same idea. When a large language model jumps straight to an answer, it can make a careless mistake. When it writes each step, every new step can build on the previous one, and errors become easier to spot.
The term comes from a 2022 Google Research paper by Jason Wei and colleagues, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models". They showed that adding worked reasoning to the examples in a prompt improved results on arithmetic, common-sense and symbolic reasoning tasks. The gains showed up mainly in very large models; smaller models often produced reasoning that sounded fine but was wrong.
Chain of thought prompting example
A chain of thought prompting example is easiest to understand by comparing a normal prompt with a CoT prompt on the same question.
Normal prompt:
Q: A canteen had 23 samosas. It sold 20 at lunch and then made 6 more. How many samosas does it have now? A:
A model answering directly might blurt out a wrong number. Now the CoT version, using one worked example (few-shot CoT):
Q: Riya has 5 pens. She buys 2 packs with 3 pens each. How many pens now? A: Riya starts with 5 pens. 2 packs of 3 pens is 6 pens. 5 + 6 = 11. The answer is 11. Q: A canteen had 23 samosas. It sold 20 at lunch and then made 6 more. How many samosas does it have now? A:
Because the example shows the working, the model copies the habit: "23 minus 20 is 3. 3 plus 6 is 9. The answer is 9." Each step is simple, so the final answer is far more likely to be right.
The example in the prompt is what few-shot prompting adds; the written reasoning is what makes it chain-of-thought.
Why does chain of thought prompting improve reasoning?
Chain of thought prompting improves reasoning because the model generates its answer one token at a time, and writing out steps gives it more intermediate text to build each next step on.
A standard LLM has no separate scratchpad. Whatever it "thinks" must appear in the tokens it writes. If the answer comes first, the model has to compress all the reasoning into one guess. If the steps come first, each step is a small, easy prediction, and the final answer is conditioned on all of them.
CoT helps most on:
- Maths word problems and percentage questions
- Logic puzzles and multi-condition rules
- Questions that combine several facts
- Code debugging, where each line needs checking
It helps little on simple lookups like "What is the capital of Kerala?" and it costs more, since longer answers use more tokens and take longer at inference time.
What is zero-shot chain of thought ("Let's think step by step")?
Zero-shot chain of thought means triggering step-by-step reasoning with a short instruction instead of worked examples.
A 2022 paper by Kojima and colleagues, "Large Language Models are Zero-Shot Reasoners", found that simply adding the phrase "Let's think step by step" before the answer improved accuracy on several reasoning benchmarks.
Q: A train leaves Pune at 9:15 am and reaches Mumbai at 12:40 pm. How long is the journey? Let's think step by step.
You get the benefit of CoT without writing examples, which makes it a combination of zero-shot prompting and chain-of-thought. I always tell students: if a model gets a maths question wrong, try adding that one line before you change anything else.
Chain of thought vs prompt chaining vs tree of thought
Chain of thought happens inside one response, prompt chaining splits a task across several prompts, and tree of thought explores multiple reasoning paths before choosing one.
| Technique | How it works | Good for |
|---|---|---|
| Chain of thought | One prompt, model writes steps then answers | Maths, logic, single questions |
| Prompt chaining | Output of prompt 1 becomes input of prompt 2, and so on | Long workflows like research, then outline, then draft |
| Tree of thought | Model branches into several ideas, evaluates them, keeps the best | Puzzles and planning with many possible paths |
Prompt chaining is common in apps and AI agents, where each stage is a separate call. Tree of thought, proposed in a 2023 paper by Yao and colleagues, needs extra code to manage the branches, so beginners rarely use it directly.
Do reasoning models still need chain of thought prompting?
Often not, because many newer reasoning models are trained to work through problems step by step on their own before answering.
Models marketed as "reasoning" or "thinking" models, such as OpenAI's o-series and DeepSeek-R1, generate a long internal reasoning process first. Some apps show a summary of it; others keep it hidden. With these models, some providers suggest keeping prompts simple and not forcing "think step by step", since the model already does a version of it.
For regular chat models, CoT prompting is still a useful tool. Details change quickly, so check the current prompting guide for the model you use.