What is temperature in an LLM?
Temperature in an LLM is a number you set before generation that controls how much randomness the model uses when it picks each next token.
A language model never "knows" the next word for sure. For every step, it calculates a probability for thousands of possible next tokens. Temperature decides how strictly the model follows those probabilities:
- Low temperature (0 to 0.3): the model almost always picks the most likely token. Answers are focused, consistent and repeatable.
- Medium temperature (around 0.5 to 0.8): a balance of accuracy and natural variety, common for chat.
- High temperature (1 and above): less likely tokens get a real chance. Output becomes more surprising, which helps brainstorming but raises the risk of nonsense.
If you are new to how models generate text one piece at a time, read what tokens are and what an LLM is first.
How does temperature work? (the simple maths)
Temperature works by dividing the model's raw scores (called logits) before they are turned into probabilities with the softmax function.
probability of a token = softmax(logits / temperature)
Dividing by a small temperature stretches the gaps between scores, so the top token dominates. Dividing by a large temperature squeezes the gaps, so many tokens end up with similar chances.
A tiny example. Say the model is completing "I drank a cup of..." and its top options are tea, coffee and water:
| Next token | Temperature 0.2 | Temperature 1.0 | Temperature 1.8 |
|---|---|---|---|
| tea | very likely | likely | slightly likely |
| coffee | rare | fairly likely | about as likely |
| water | almost never | possible | about as likely |
At temperature 0 most systems simply pick the single most likely token every time, which is called greedy decoding. That is why low temperature feels "safe" and high temperature feels "creative".
What is the range of temperature in an LLM?
The usual temperature range in LLM APIs is 0 to 2, though the exact limits depend on the provider.
For example, OpenAI's API accepts values from 0 to 2, while Anthropic's Claude API accepts 0 to 1. Chat apps like ChatGPT, Claude and Gemini mostly hide the setting from regular users and pick a sensible default for you. Developers set it in code or in tools like an API playground.
Because ranges and defaults change, always check the current documentation of the model you use. A temperature of 1.0 on one platform is not guaranteed to behave exactly like 1.0 on another.
What temperature should you use?
Use a low temperature for tasks with one right answer and a higher temperature for tasks that benefit from variety.
- Code, maths, data extraction, classification: 0 to 0.2. You want the same correct answer every time.
- Factual Q&A, customer support, RAG answers: 0 to 0.4. Stay close to the source text and reduce hallucination.
- Everyday chat and explanations: 0.5 to 0.8. Natural, not robotic.
- Brainstorming, taglines, story ideas: 0.8 to 1.2. More surprising options to choose from.
- Above about 1.3: rarely useful outside experiments; text often becomes incoherent.
My rule for students building their first app: start at 0.2, check your outputs, and only raise it if the answers feel stiff. Most bugs I see in beginner projects come from creativity nobody asked for.
Temperature is also a favourite question in generative AI interviews, usually followed by "does it make the model more accurate?" (it does not).
Temperature vs top-p vs top-k
Temperature reshapes all the probabilities, while top-p and top-k limit which tokens are allowed at all.
- Top-k keeps only the k most likely tokens (say, the top 40) and samples among them.
- Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p (say, 0.9) and samples among them.
- Temperature changes how evenly probability is spread across whichever tokens remain.
They can be combined, but changing several at once makes results hard to reason about. A common piece of advice from model providers is to adjust either temperature or top-p, not both, and to change one setting at a time while testing.
Common myths about LLM temperature
The biggest myth is that a lower temperature makes a model more accurate.
- Myth: low temperature means correct answers. It only makes the model more consistent. If the model is wrong, it will be consistently wrong. Accuracy comes from better prompts, grounding with RAG and better models.
- Myth: temperature 0 is perfectly deterministic. It is close, but tiny differences in hardware and batching can still change outputs occasionally, so do not build systems that assume identical responses.
- Myth: high temperature means more intelligence. It means more randomness. Sometimes that surfaces a better idea; often it adds errors.
Treat temperature as a style dial, not a quality dial.