Beyond Chinchilla: How Do You Balance Training and Inference Costs?

by
Table of contents

🌏 閱讀中文版本

A PM forwarded a paper. The note said: “Is this our next move?”

You open the cost report. Inference is 45% of the total.

You stare at the Chinchilla formula and realize it does not tell you how to price this part of the equation.

When inference demand is very high, for example around ~10^11 tokens, training a smaller model that has seen more data can be cheaper than strictly following Chinchilla’s compute-optimal rule.

That sounds counterintuitive. For the past two years, the industry has followed a simple principle: scale model size and data together at a fixed ratio. But once you add inference cost to the full equation, that golden ratio stops being the whole story.

This 2024 paper from MosaicML and Databricks adds the missing piece to scaling laws: inference is not a side effect of training, it is a variable that shapes model size.

Where Chinchilla Applies: Optimizing Training Compute

To understand this paper, we need the prequel first.

In 2020, Kaplan et al. published the first scaling law paper. They found that model performance follows a log-linear relationship with parameter count and data size. But the formula was still rough. It did not give concrete optimization guidance.

In 2022, Hoffmann et al. published the now-famous Chinchilla paper. They systematically trained many models at different scales and reached a conclusion that reshaped the field:

The Chinchilla rule: for a given compute budget, there is a “golden ratio” between parameters and training tokens, about 20 tokens per parameter.

In simple terms, Chinchilla tells engineers: “If you have a $1 million training budget, do not just add parameters. Increase data too, and keep the ratio tight.”

That rule is highly precise for the training stage. But one assumption creates a blind spot: it treats inference cost as negligible.

Chinchilla optimizes for one goal: the highest model quality under fixed training compute. It does not ask how often the model will be deployed, or how many users will call it after training.

That is the gap this new paper addresses.

The Core Mechanism: How Inference Demand Reverses the Optimum

This paper, which I will call the “new paper,” makes a simple mathematical adjustment:

Total cost = training cost + inference cost

In the Chinchilla frame, total cost is approximately training cost. In the new paper, total cost = training cost + (number of inference calls × compute cost per inference).

When inference volume is low, the new paper reaches almost the same conclusion as Chinchilla. But when inference demand reaches a high-volume regime, for example ~10^11 tokens, the relationship flips.

The Marginal Cost View

Imagine you are planning cloud infrastructure.

  • Chinchilla’s logic, one-time investment
    You have a fixed total budget for training compute. To get the best performance on one task, training, you choose the model size that fits that objective. That is reasonable during training.

  • The new paper’s logic, long-term serving cost
    You realize the model does not run once. It handles millions of API requests a day, 24/7.
    At that point, the marginal compute, memory bandwidth, and serving cost of a larger model start to accumulate fast.
    The best answer becomes: choose a smaller, more efficient model, then train it longer to make up for the capacity gap.

Why? To reach the same model quality, a smaller-than-Chinchilla model needs more training compute, because it must see more data to compensate for fewer parameters. But if inference volume is high enough, the long-term inference savings from the smaller model exceed the extra training cost up front.

The causal chain looks like this:
Very high inference demand → inference cost dominates total cost → per-request compute must come down → choose a smaller model → to preserve target quality, add extra training compute by overtraining on more data → in the end, inference savings > extra training spend.

That is why the paper is called “Training Smaller, Longer.”

What Actually Matters: Numbers and Validation

This paper is not just theory. The authors trained 47 models and tested how scaling laws behave in extreme regimes.

Here are three results that matter for real decisions.

1. The Golden Ratio Moves with Inference Demand

Chinchilla says the golden ratio is 20 tokens per parameter. The new paper finds that once inference demand reaches ~10^11 tokens, the best ratio rises sharply.

The concrete example:

For a model with Chinchilla-quality performance at the 7B scale, if expected inference demand is 10^11 tokens, the better strategy is to train a 6B model on 1.18x the data Chinchilla would suggest.

Notice the detail: the model gets smaller, 7B to 6B, but the data gets larger, 1.18x. That sounds contradictory. But the data shows this gives the best total cost once you combine training and inference.

2. Model Quality Keeps Improving Even at Extreme Ratios

Chinchilla’s data mostly sits in the standard range, up to 100 tokens per parameter. The new paper pushes the ratio much further, up to 10,000 tokens per parameter, and finds this:

Model quality does not hit a saturation point. It continues to improve.

That means if you are willing to spend more training time, you can use a smaller model and still reach similar quality. This gives real support to the “small model, more data” strategy.

3. The Limits of Extrapolation

The paper also points out that if you fit scaling law coefficients using only standard Chinchilla data, low token-to-parameter ratios, you will overestimate the benefit of extra data in the extreme-ratio regime.

In plain English: Chinchilla’s formula becomes less reliable when you extrapolate into high-ratio territory. If you use it directly for a high-inference product, you are likely to train a model that is too large and under-trained on data, which pushes inference cost up quickly.

Your Decision: Boundaries and Tradeoffs

This paper gives a clear decision matrix. But “smaller, longer” does not fit every workflow.

Good fit

  • High-frequency serving models: your model handles millions of API calls or powers a real-time recommendation system.
  • Cost-sensitive products: inference cost is already a major OpEx line item.
  • Edge deployment: the model must run on resource-constrained devices such as phones or IoT hardware.

Less suitable fit

  • Low-frequency, high-complexity tasks: scientific research or long-form generation, for example. These tasks may depend more on zero-shot generalization, long-context integration, or complex reasoning. If quality is mostly limited by those factors, you need a separate evaluation.
  • Extremely constrained training compute: if you do not even have enough compute to train a small model from scratch, then you can only rely on fine-tuning pre-trained models rather than training from scratch.

The Tradeoff: Time vs. Space

At its core, “train smaller, longer” means trading training time for inference efficiency.

  • Cost: longer training cycles, larger datasets, and more complex data pipelines.
  • Return: much lower inference cost over time, with faster model responses.

This is not a question of right or wrong. It is a question of choice.
If your product is one-off, for example a single report generation flow, Chinchilla still works well.
If your product is a continuously serving system, such as a chatbot or translation API, this paper is a good reason to re-examine the relationship between model size and data volume.

Why Do So Many Teams Still Choose Larger Models?

There is a finance-process reason behind that choice.

Training cost is a one-time line item. It is easier to approve. Inference cost is ongoing OpEx. It is easier to miss in a monthly report.

That difference in visibility shapes decisions. Many teams choose a larger model because it reduces the approval friction up front.

If you understand the approval rhythm, you can see where the real risk sits.

Follow the risk.
Vendors and adopters carry different kinds of risk. The adopter has to manage long-term inference OpEx.
Once inference volume gets high enough, the risk shifts from “training may not meet expectations” to “operational cost may keep growing.”

Verifiable Sources and Limits

The paper’s core claim is based on empirical results from 47 trained models and a derived compute optimization formula.

Key references:

Limits noted by the authors:

  1. MFU differences: hardware utilization during inference is usually lower than during training, and the gap depends on hardware and workload. The paper includes this difference in the cost model, but real hardware choices, such as GPU type and network bandwidth, will still affect results.
  2. Data quality: the conclusion assumes data quality can be maintained. If data quality falls, the benefit changes. In practice, finding more high-quality, low-bias data is difficult, and the risk of data contamination rises with larger datasets.
  3. The market signal isn’t clear yet: as hardware improves, especially dedicated inference chips, inference cost may decline. If that happens, the optimal ratio may move again.

Conclusion

Inference volume is worth bringing into the earliest scaling-law assumptions.

Chinchilla gave you a ruler. But that ruler measures training cost, not total cost. Once you include inference, does it still measure the right thing?

The earlier inference volume enters planning, the more controllable model-size decisions become.

Is your product a one-off report, or a continuously serving system? That answer shapes where model size and training-data ratio should move. Is this a calculation worth doing before the next model decision?