Neural Scaling Laws

Neural scaling laws describe the predictable, power-law relationships between a model’s performance and three key resources: training compute, dataset size, and model parameter count. Understanding these laws has fundamentally reshaped how AI labs train large language models.

The Kaplan Scaling Laws (2020)

OpenAI researchers published the first comprehensive empirical study of neural scaling in 2020 (Kaplan et al.). Their findings showed that loss on language modeling tasks follows a smooth power law across many orders of magnitude:

$$L(N) \propto N^{-\alpha_N}, \quad L(D) \propto D^{-\alpha_D}, \quad L(C) \propto C^{-\alpha_C}$$

Where:

• $N$ = number of model parameters
• $D$ = dataset size (tokens)
• $C$ = total training compute (FLOPs)
• $\alpha$ values are empirically measured exponents (~0.07–0.095)

Key Findings

• Model size matters most when compute is the bottleneck. The Kaplan paper recommended scaling parameters aggressively while under-training on data.
• Performance improves smoothly and predictably — no sudden walls or diminishing returns at the scales studied.
• Compute-optimal frontier: for a fixed compute budget, there is an optimal balance between model size and number of training tokens.

The Chinchilla Scaling Laws (2022)

DeepMind’s Hoffmann et al. revisited Kaplan’s conclusions with a much larger range of controlled experiments. Their paper — colloquially called “Chinchilla” — reached a different conclusion:

For compute-optimal training, model size and training tokens should scale roughly equally.

The Chinchilla formula implies that for every doubling of model parameters, you should also roughly double the number of training tokens. Kaplan-era models (like GPT-3 at 175B parameters trained on ~300B tokens) were significantly undertrained according to this analysis.

The Chinchilla-Optimal Rule of Thumb

A model trained compute-optimally should see approximately 20 tokens per parameter:

$$D_{\text{optimal}} \approx 20 \times N$$

This led to models like Chinchilla 70B (trained on 1.4 trillion tokens) dramatically outperforming much larger models trained on less data.

The Three Scaling Axes

These three axes are not independent — doubling compute can be spent on a bigger model, more data, or both. Scaling laws tell you the most efficient allocation.

Data Scaling Challenges

Modern frontier models have largely exhausted high-quality internet text. This creates a data wall where continued scaling along the token axis becomes increasingly difficult:

• Common Crawl and other web corpora have been used multiple times (multi-epoch training).
• Synthetic data (generated by other LLMs) is being used to extend effective dataset size.
• Curated high-quality data (books, code, scientific papers) may be worth orders of magnitude more than raw web text — implying data quality has its own scaling relationship.

Emergent Abilities

One of the most striking observations in large model training is the appearance of emergent abilities — capabilities that seem to appear abruptly at certain scale thresholds rather than improving gradually.

Examples include:

• Multi-step arithmetic — nearly absent below ~10B parameters, then suddenly functional.
• Chain-of-thought reasoning — emerges around 10–100B scale.
• In-context learning — few-shot prompting becomes reliable only at large scale.

The Debate

Some researchers (Schaeffer et al., 2023) argue that emergent abilities are an artifact of the evaluation metric — using discontinuous metrics like exact-match accuracy on tasks creates artificial threshold effects. Under continuous metrics, the same abilities appear to scale smoothly.

The debate highlights a key limitation of scaling laws: they were derived on aggregate perplexity / loss, but real-world capability often depends on binary task success metrics.

Inference-Time Scaling

A newer paradigm — championed by OpenAI’s o-series and DeepSeek-R1 — shows that scaling inference compute (thinking tokens at test time) yields its own power-law improvements on reasoning tasks. This suggests a second axis of scaling:

• Pre-training compute → general capability
• Inference compute → reasoning depth on specific problems

This has led to the concept of “thinking budgets” where harder problems receive more reasoning tokens.

Implications for Model Training Strategy

Understanding scaling laws has direct practical implications:

1. Don’t over-invest in parameters alone. A smaller model trained on much more data often outperforms a larger undertrained model at the same compute cost.
2. Predict performance before training. Running small-scale pilots on a log-linear compute range lets labs extrapolate final loss before committing to full runs.
3. Multi-epoch training is possible. Repeating data multiple times hurts less than leaving compute on the table, especially with diverse, high-quality datasets.
4. Data quality has a multiplier effect. Filtering low-quality data can shift the effective data scaling exponent.

Current Frontiers

• Beyond loss to capabilities: Deriving scaling laws directly for downstream task performance, not just perplexity.
• Multimodal scaling: How do scaling laws interact when training on text, images, audio, and video jointly?
• Architecture-specific laws: Do transformer-alternative architectures (SSMs, hybrid models) obey the same exponents?
• Post-training scaling: How do RLHF and instruction tuning interact with pre-training compute?

Neural scaling laws remain one of the most powerful predictive tools in AI development — providing a rare form of empirical predictability in an otherwise unpredictable field.