Kaplan 2020: Scaling Laws and Their Limits

🌏 閱讀中文版本

The loss curve starts flattening after epoch 50.

You’re watching GPU utilization, and doing the math in your head: should the next card be bigger, or should you go find more data?

That pause comes from a core tradeoff: when compute is limited, how should we allocate resources to get the most out of performance?

In 2020, OpenAI published a widely cited paper that offered a predictable loss-scaling framework. But two years later, real-world practice showed that its recommendation needed a more precise boundary of applicability.

Understanding this history is not about academic trivia. It’s about avoiding old assumptions in your next round of resource allocation.

Core Idea: A Shift From Model Scale to Data Scale

Kaplan 2020 gave the industry a map, but that map reflected the tradeoffs between compute and data costs at the time.

The assumption then was simple: data collection was expensive, and high-quality data was hard to get. Under that constraint, the Kaplan team observed that within their experimental range, increasing model size (N) reduced loss more effectively than increasing data volume (D).

That study shaped how the industry trained models for the next few years. But one key inference in that work needed a clearer boundary later on.

Then in 2022, the Chinchilla paper introduced a compute-optimal configuration based on new experiments, and re-estimated the balance between model size, data volume, and compute.

Why Did Early Decisions Favor Bigger Models?

When LLMs were just starting to gain momentum, engineers faced a large uncertainty: when we scale up training compute, how exactly does model performance change?

Intuition says a bigger compute budget should produce a stronger model. But what does the marginal gain look like? Is it linear? Exponential? Or does it hit a ceiling quickly?

Before the market signal and measured results became clear, it was hard to answer practical tradeoffs like these:

1. Resource allocation: should the budget go to more GPUs, or to collecting more data?
2. Model choice: should you train a 175B-parameter giant, or a leaner 7B model?
3. Cost estimation: how far do you need to train before the model becomes genuinely useful?

Kaplan tried to answer those questions with data. But that answer was built on the assumption that data was extremely expensive.

The Core Mechanism: Why Did “Big Model, Less Data” Become the Default?

The central finding of Kaplan 2020 was this: model loss has a stable power-law relationship with model size (N), data volume (D), and compute (C).

In plain English: in the unsaturated regime, scaling compute, data, and parameters together produces a predictable drop in loss.

The Key Inference: Sample Efficiency

The Kaplan team found something important in their analysis: as the model gets larger, it learns more efficiently from each data point.

That means a larger model can extract more information from the same dataset. Intuitively, that seems to support the idea that bigger is better.

So Kaplan’s recommendation was: under a fixed compute budget, prioritize model size (N), because larger models are more sample-efficient.

That inference fully holds within their experimental setting. But once you shift the lens to compute-optimal training, putting too much weight on parameters while underweighting data can reduce overall training efficiency.

When data is repetitive, or lacks quality and coverage, adding parameters may not improve generalization. It’s like adding more analysts when no new intelligence is coming in. More people just over-interpret old signals. They don’t create new insight.

The Result That Really Matters: Chinchilla’s Correction

Kaplan did not get the math wrong. The paper derived a larger-model allocation under the data costs and experimental setup of its time.

Then in 2022, DeepMind published Chinchilla. Using new training runs and scaling analysis, the paper re-estimated the compute-optimal configuration and revised Kaplan’s conclusion.

Their finding was clear: Kaplan’s data points were sitting in a suboptimal region.

When they increased data volume (D) substantially and reduced model size (N) accordingly, they found this:

1. Chinchilla (70B parameters, about 1.4T tokens) outperformed GPT-3 (175B parameters, 300B tokens) on multiple benchmarks.
2. Key ratio: Chinchilla found that model size and data volume should stay around a 1:20 ratio (about 20 tokens per parameter). In Chinchilla’s setup, that became an empirical compute-optimal reference point.

There are two different ideas here, and they should not be conflated: “scale model and data together as compute grows” is a dynamic balance, while “about 20 tokens per parameter” is a concrete reference ratio.

In other words, Kaplan’s “big model, less data” recommendation reflected the data availability and estimation methods of that era. The industry could not gather enough data, so many teams treated larger models as the answer.

A Concrete Comparison

Chinchilla’s real message was this: the importance of data had been underestimated; model size and data volume need to be balanced together. If you increase model size without increasing data, the configuration can drift away from the compute-optimal point, and both data efficiency and generalization can fall short of a more balanced setup under the same compute budget.

Your Decision: Boundaries and Tradeoffs

What does this history mean for your decisions today?

1. Check Whether Data Is the Bottleneck First

If you’re planning a new training project, make sure data quality and data volume are in place first, rather than expanding parameter count by default.

• A fit for this situation: you have enough compute, but limited data sources.
• Tradeoff: when data quality, architecture, and training recipe are comparable, a medium-sized model (say 7B-13B) that is fully trained is often more effective in practice than a very large model trained on insufficient data.

2. The Boundary on Compute Allocation

Kaplan’s power law still holds, but the context for applying it has changed.

• When data is abundant: increasing model size can still improve performance, but loss falls by a power law, so marginal gains diminish.
• When data is limited: simply making the model larger pushes the configuration away from the compute-optimal point. Improvements in generalization will no longer track your compute spend proportionally.

3. What This Means in Practice

• Model comparison: the more useful comparison is the combination of parameter count, training tokens, and data-processing strategy. A 7B model trained on 100B+ tokens can be stronger than a 13B model trained on only 10B tokens.
• Cost structure: the share of total cost taken up by data processing is worth estimating on its own.

Sources and Limits

Kaplan’s inference made sense in its original context because data collection costs were extremely high. Chinchilla’s correction is not perfect either, because it assumes data can keep scaling. In practice, the marginal cost of high-quality data rises over time.

At very high data volumes, the market signal isn’t clear yet. The best future configuration will still depend on deduplication techniques, diversity, and the evaluation tasks you actually care about.

Your Next Step: Take Stock of Your Resources

1. Estimate your training data volume: if you’re training a model, calculate the N:D ratio. If tokens per parameter sits well below the compute-optimal reference, check the data gap first.
2. Include training data volume in model comparisons: when comparing models, don’t stop at parameter count. Also look at training data volume and deduplication strategy.

Kaplan gave you a map, but the terrain has changed. So how much data do you actually have on hand now?

This isn’t about right or wrong. It’s about where you want to put your compute budget into parameters or into data. That’s your call.

Sources

• Scaling Laws for Neural Language Models (Kaplan et al., 2020) — Introduced power-law scaling laws and the larger-model recommendation
• Chinchilla: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022) — Proposed an optimized balance between model size and data volume