Scaling Laws

Scaling laws are the empirical reason the field bet on size. They say that model loss falls as a smooth, predictable power law in three quantities — parameters, data, and compute — across many orders of magnitude. That predictability is what lets a lab justify spending tens of millions of dollars on a single training run before seeing a single result.

The Power Law

The Kaplan et al. (2020) result: test loss scales as a power law in model size (N), dataset size (D), and compute (C), as long as you're not bottlenecked on the other two. Plotted on log-log axes, loss-versus-compute is a near-straight line. No sign of a wall — just diminishing, but continuing, returns.

The practical punchline: performance is more a function of scale than of clever architecture tweaks. Most of the gains from GPT-2 to GPT-4 came from scale.

Compute-Optimal Training (Chinchilla)

Kaplan's original recipe over-weighted parameters. The DeepMind Chinchilla paper (2022) corrected it: for a fixed compute budget, you should scale parameters and training tokens roughly in proportion — about 20 tokens per parameter.

This reframed the field:

• A 70B model trained on 1.4T tokens (Chinchilla) beat a 280B model trained on far fewer (Gopher), using the same compute.
• "Compute-optimal" is not the same as "deployment-optimal." If you'll serve a model to billions of requests, it's worth over-training a smaller model past the Chinchilla point — you pay more in training to save far more in inference. This is why Llama and most production models are small-but-over-trained.

Why It Matters to an Engineer

You rarely train a frontier model. But scaling laws shape decisions you make constantly:

• Model selection — a smaller model trained on more tokens often beats a larger, under-trained one. Parameter count alone is a poor proxy for capability.
• Fine-tuning budgets — the same loss-vs-data curve governs how much your fine-tuning gains from each additional labeled example. Returns diminish predictably; you can estimate when more data stops being worth it.
• The over-training trend — it's why a 7B–8B model today can match a 70B model from two years ago, and why "small model, served cheaply" is a viable product strategy.

Where Scaling Laws Bend

• Data is finite. High-quality text is being exhausted. Synthetic data, multimodal data, and better data curation are how labs keep feeding the curve.
• Emergent abilities — some capabilities (multi-step reasoning, instruction following) appear abruptly at scale rather than improving smoothly. Whether these are real phase transitions or artifacts of how you measure is still debated.
• Inference-time scaling — reasoning models added a second axis: spend more compute at inference (longer chains of thought) rather than at training. This is its own scaling curve, and it's where a lot of recent gains live.

What to Watch For

Don't read "loss goes down" as "my product gets better." Scaling laws are about next-token loss on a held-out set. The mapping from loss to downstream task performance you care about is noisy and task-dependent. Always validate on your own evals — the curve tells you what to expect from the base model, not whether it solves your problem.