Artificial Wasteland · the one knob, operated
The Temperature Dial
Every model you talk to has one knob for how surprising it's allowed to be. It is a single number, and it does exactly one thing to the arithmetic. Here it is, on a real model — turn it, and watch.
When a language model writes the next word, it does not choose a word. It produces a score for every entry in its vocabulary at once — for GPT-2, all 50,257 of them — and then those scores are turned into probabilities and one is drawn. Temperature is the number that sits between the scores and the probabilities. (Strictly, those 50,257 entries are tokens, not words — mostly whole words, some word-pieces and punctuation, each carrying any leading space; you'll see a few like ␣1 or : in the bars below. I'll say "word" loosely from here on.) It is not a mood, not a creativity setting, not a personality. It is a divisor. Everything below is computed live from a real GPT-2's actual next-word scores; nothing here is illustrated or approximated.
—
␣ marks a leading space — GPT-2's tokens include it, so "␣the" and "the" are different tokens.
Slide the dial toward 0 and the tallest bar swallows the rest — the model has, in effect, one choice. Slide it up and the bars flatten as probability leaks into the long tail of thousands of words the model barely considered. Watch the order of the bars: it never changes. Temperature cannot promote a word past the one above it — it only decides how steep the cliff between them is. And the sampling marks (draw a few hundred) show the payoff: at low T every draw is the same word; at high T the draws scatter into words the model rated near-hopeless.
One divisor, before the softmax
The raw scores are called logits. To turn them into probabilities the model uses the softmax function, which exponentiates each score and normalises so they sum to one. Temperature does its whole job in one place: it divides every logit by T before the exponential.
That single division explains everything the dial does:
- T = 1 — divide by one, change nothing. This is the model's own distribution, the voice it was trained to have.
- T → 0 — dividing by a tiny number blows the gaps between scores up enormously, so after the exponential the single largest logit takes essentially all the probability. This is greedy decoding: always the top word.1
- T → ∞ — dividing by a huge number crushes every score toward zero, so after the exponential they're all nearly equal: the distribution flattens toward uniform, pure noise over the whole 50,257-word vocabulary.
The word "temperature" is not a metaphor borrowed loosely. This is exactly the Boltzmann distribution from statistical mechanics — the probability that a physical system sits in a state of energy E is proportional to exp(−E / kBT), with T the literal temperature. Set each energy equal to the negative logit and you recover the formula above line for line.2 A high-temperature gas rattles through all its states; a cold one settles into the lowest. A hot model rattles through its whole vocabulary; a cold one settles onto the single likeliest word. Same equation, same behaviour.
Why the entropy rises — a fact, not a claim
The "effective choices" readout is 2H, where H is the distribution's Shannon entropy in bits — roughly, how many words the model is genuinely spreading its bet across. It climbs monotonically with T, and that isn't an observation about GPT-2; it's a theorem. Writing β = 1/T, the entropy obeys
where Var(z) is the variance of the logits under the current distribution. A variance is never negative, so raising T can never sharpen the distribution — only soften it. (The one exception: if every logit were identical there'd be zero variance and nothing to soften. Real logits are never identical.)3 This is the same statement as "heat capacity is non-negative" in thermodynamics — the physics and the arithmetic are the same object.
The two ways it breaks
The dial has a comfortable middle and two cliffs. To make the cliffs concrete, here is a single real GPT-2 continuing one fixed prompt — "The old lighthouse keeper wrote in his logbook:" — at six temperatures, from the same random seed, with pure temperature sampling and nothing else (no top-k, no top-p — so what you see is temperature's effect alone). Only the dial moved between these.
The bottom of the range is not "boring but safe" — it's the repetition trap. Greedy decoding keeps taking the single most likely next word, and the most likely word after "I have never seen anything like it." is the start of "I have never seen anything like it." The model falls into a loop and can't climb out, because climbing out would require taking a word it rated lower. This is the well-documented failure of maximisation decoding.4
The top of the range is the degeneration trap. GPT-2's vocabulary has 50,257 words; on any given step, tens of thousands of them share a small slice of probability. That slice is the "unreliable tail." Turn the temperature up and you inflate the tail until sampling starts pulling words from it — and the sentence dissolves into "Photoshop tenancy" and "F196 Ball joy skipped." High temperature doesn't make the model creative; it makes it drink from the part of the distribution it was least sure about.5
What the dial is not
Temperature is not the only knob
Temperature reshapes the whole distribution — every word's probability changes, but no word is ever removed. The other common controls instead truncate it before sampling:
- top-k keeps only the k highest-scoring words and throws the rest away.6
- top-p (nucleus sampling) keeps the smallest set of words whose probabilities add up to p — a set that shrinks when the model is confident and grows when it isn't.5
Real deployments usually combine several of these, and the order in which they're applied changes the result. So "I set temperature to 0.8" rarely tells the whole story of how a given API was sampling.
T = 0 is not guaranteed to be deterministic
Setting T = 0 makes the sampling rule deterministic — always take the top word. But the logits feeding that rule are computed on GPUs in large batches, and floating-point addition isn't associative: (a+b)+c need not equal a+(b+c). When the server's batch size shifts with load, the arithmetic reorders and the logits come out slightly different, which can flip the argmax. In one published test, a large model at temperature 0 produced 80 distinct completions across 1,000 identical requests. The math of T = 0 is deterministic; a production endpoint is not promised to be.7
Temperature cannot fix a wrong belief
Try the "Two plus two equals" prompt above and turn the dial to 0. GPT-2's top word is "one" — confidently, at 33% — and it is confidently wrong. That prompt is also the lowest-entropy, most certain of the five. The dial only decides how far down the model's own ranking you are willing to gamble; it cannot move a right answer up the list if the model didn't put it there. A small 124-million-parameter model like this one is often surest exactly where it's most mistaken, and no temperature setting repairs that. It is a good thing to keep in the front of your mind about every model you use.
So: one number, one division, done before the exponential. Below it lies a loop the model can't escape; above it, noise it can't resist. Between them is every conversation you've ever had with a machine — and now you know precisely which way the knob was turned.