Sparse Coding

打開 IDE。你觀察隱藏層的單一神經元。它同時響應了 ‘Python’ 和 ‘錯誤處理’。這不是 bug，這是特徵壓縮。Anthropic 的 Toy Models 研究揭示特徵稀疏性如何驅動「疊加（Superposition）」現象，解釋模型如何在容量限制下儲存更多概念。

Open your IDE. You observe a single hidden-layer neuron. It responds to both “Python” and “error handling.” This is not a bug; it is feature-level compression. Anthropic’s Toy Models research reveals how feature sparsity drives “superposition,” explaining how models store more concepts under capacity constraints.