Sampling & Decoding

A language model doesn't output text — it outputs a probability distribution over the next token, every step. Decoding is the algorithm that turns that distribution into the token you actually get. The same model can be a reliable extractor or a creative writer depending entirely on these settings, and getting them wrong is one of the most common causes of "the model is broken" reports that aren't actually the model's fault.

Temperature

Temperature rescales the logits before the softmax. It's the master volume knob for randomness.

• Low (0–0.3) — sharpens the distribution toward the highest-probability tokens. Use for extraction, classification, code, structured output, anything where you want the same answer every time.
• Medium (0.7–1.0) — the default conversational range. Balanced.
• High (>1.0) — flattens the distribution, surfacing unlikely tokens. Use for brainstorming and creative variety. Push too far and output degrades into noise.

Temperature 0 is near-deterministic, not guaranteed deterministic — batching, hardware, and MoE routing introduce small nondeterminism even at temp 0.

Top-k and Top-p (Nucleus)

These truncate the distribution before sampling, so the long tail of garbage tokens can't be picked even by chance.

• Top-k — keep only the k most likely tokens, renormalize, sample. Simple but rigid: k=40 is too many in confident contexts and too few in uncertain ones.
• Top-p (nucleus) — keep the smallest set of tokens whose cumulative probability ≥ p (e.g. 0.9). Adapts to context: few tokens when the model is confident, more when it's not. Generally preferred over top-k.

The usual recipe: set temperature for randomness, top-p (~0.9–0.95) as a safety net against the tail. Don't aggressively tune both at once — they interact.

Other tail controls you'll see: min-p (keep tokens above a fraction of the top token's probability), and repetition / frequency / presence penalties to discourage loops and verbatim repetition.

Greedy and Beam Search

• Greedy — always take the single highest-probability token. Equivalent to temperature 0. Deterministic, but can get stuck in repetitive or locally-optimal phrasing.
• Beam search — track several candidate sequences in parallel and keep the highest-probability sequence. Good for tasks with one right answer (translation, transcription); bad for open-ended generation, where it produces bland, repetitive text. Rarely used for chat.

Constrained / Structured Decoding

When you need valid JSON or a value from a fixed set, don't hope the model complies — constrain it. Structured decoding masks the logits at each step so only tokens that keep the output grammar-valid are allowed (via a JSON schema, regex, or context-free grammar). Libraries like Outlines, XGrammar, and most inference servers support this. It guarantees parseable output and often lets a smaller model hit formats a larger one would miss. (See also Prompt Engineering → Structured Output.)

Speculative Decoding

A throughput trick, not a quality one: a small "draft" model proposes several tokens, the big model verifies them in a single forward pass, accepting the ones it agrees with. Output is identical to normal decoding from the big model, but latency drops because you amortize the expensive forward passes. Handled at the inference-engine layer, not by you — but it's why some endpoints are dramatically faster than their model size suggests.

What to Watch For

The single biggest mistake: leaving the default temperature (~0.7–1.0) on a task that needs determinism. If your evals are flaky, your JSON sometimes fails to parse, or "the same prompt gives different answers," check your decoding settings before you blame the prompt or the model. For anything you'll grade or parse, start at temperature 0 plus structured decoding, and only add randomness where variety is the goal.