Chinchilla Scaling Laws: 算力預算下的模型與資料比例優化
算力預算有限時，該追求大模型還是大資料？透過 Chinchilla 論文數據，解析 1:20 參數與 Token 的最優比例，為 LLM 訓練提供決策依據。
Scaling Laws
Chinchilla: Model/Data Ratios on a Budget
When compute is limited, should you buy a bigger model or feed it more data? Chinchilla gave a counterintuitive answer in 2022: at a fixed training budget, a 70B model trained on 1.4T tokens outperformed a 280B model trained on 300B tokens, using the same compute. That points to an approximate 1:20 parameter-to-token ratio in that setup. If you’re planning an LLM training run, this paper offers a practical way to think about model size, data volume, and where your next dollar creates more learning.
Kaplan 2020 論文：Scaling Laws 的演進與情境邊界
2020 年 OpenAI 提出模型大小與資料量的冪次律，建議『大模型、少資料』。兩年後 Chinchilla 修正 compute-optimal 配置，讓資料量的重要性重新被看見。拆解 Scaling Laws 演進，以及它如何影響你現在的訓練決策與成本結構。
Kaplan 2020: Scaling Laws and Their Limits
In 2020, OpenAI proposed a power-law relationship between model size and data volume, which led many teams toward a simple idea: bigger models, less data. Two years later, Chinchilla revised that picture with a compute-optimal view, showing that data volume mattered far more than many people had assumed. This matters if you’re deciding whether the next gain should come from more GPUs, a larger parameter count, or a larger corpus. The history of scaling laws is not just academic. It changes how you think about training efficiency, model selection, and where your cost structure really sits.
超越 Chinchilla：如何在訓練與推論成本之間取得平衡？
當推論需求極高時，遵循 Chinchilla 法則反而更貴。本文解析如何透過「小模型、大資料」策略，在訓練與推論的總成本中找到新的平衡點。
Beyond Chinchilla: How Do You Balance Training and Inference Costs?
When inference demand is extremely high, following the Chinchilla rule can actually be more expensive. This article explains how a “small model, more data” strategy can find a new balance point in the total cost of training and inference.