LLM

Surpassing RAG with 20% Memory: ASM Latency Optimization on the NBA Benchmark

by

Long-context inference faces two constraints at once: memory use grows with the prefix, while latency rises at every decode step. Attention-State Memory (ASM) offers a training-free alternative by externalizing precomputed attention states into a lightweight lookup-based memory. On the NBA Benchmark, it exceeded full-attention RAG performance using about 20% of the memory. This article explains ASM’s hierarchical lookup and online-softmax merge, then maps the boundaries that matter in deployment: query-distribution stability, prefix updates, offline construction, and codebook-size tuning.

Chinchilla: Model/Data Ratios on a Budget

by

When compute is limited, should you buy a bigger model or feed it more data? Chinchilla gave a counterintuitive answer in 2022: at a fixed training budget, a 70B model trained on 1.4T tokens outperformed a 280B model trained on 300B tokens, using the same compute. That points to an approximate 1:20 parameter-to-token ratio in that setup. If you’re planning an LLM training run, this paper offers a practical way to think about model size, data volume, and where your next dollar creates more learning.

Kaplan 2020: Scaling Laws and Their Limits

by

In 2020, OpenAI proposed a power-law relationship between model size and data volume, which led many teams toward a simple idea: bigger models, less data. Two years later, Chinchilla revised that picture with a compute-optimal view, showing that data volume mattered far more than many people had assumed. This matters if you’re deciding whether the next gain should come from more GPUs, a larger parameter count, or a larger corpus. The history of scaling laws is not just academic. It changes how you think about training efficiency, model selection, and where your cost structure really sits.