用 20% 記憶體超越 RAG：ASM 在 NBA Benchmark 的延遲優化
長上下文推理面臨記憶體與延遲的雙重瓶頸。本文解析 Attention-State Memory (ASM) 如何透過外部化預計算狀態，在 NBA Benchmark 等特定場景下，以約 20% 的 RAG 記憶體佔用實現性能超越，並探討其適用邊界。
Inference Optimization
Surpassing RAG with 20% Memory: ASM Latency Optimization on the NBA Benchmark
Long-context inference faces two constraints at once: memory use grows with the prefix, while latency rises at every decode step. Attention-State Memory (ASM) offers a training-free alternative by externalizing precomputed attention states into a lightweight lookup-based memory. On the NBA Benchmark, it exceeded full-attention RAG performance using about 20% of the memory. This article explains ASM’s hierarchical lookup and online-softmax merge, then maps the boundaries that matter in deployment: query-distribution stability, prefix updates, offline construction, and codebook-size tuning.