Toy Models of Superposition: Feature Compression in Neural Networks

by
Table of contents

🌏 閱讀中文版本

Open your IDE. Filename: model_weights.bin.

Using a hook, you inspect a single hidden-layer neuron. It responds to both “Python” and “error handling.”

Your intuition says this should be messy. Under the convenient assumption used in some interpretability analyses, one neuron should map to one concept. Now two unrelated concepts occupy the same dimension. It looks as if the model has broken down. We call this “polysemanticity.”

But look at it differently: this is not a bug. It is feature-level compression, not compression of parameters or file size.

Anthropic’s Toy Models research reveals a counterintuitive fact: this is not a design flaw. Under particular sparsity and capacity constraints, the model learns a non-orthogonal representation strategy that allows limited interference. We call it “superposition.”

Understanding this mechanism is one key to opening the black box.

Why Does Your Neural Network Run Out of Room?

A dimensional bottleneck forces a model to give something up. Feature sparsity gives it room to work around that limit.

In traditional linear algebra, if you have N dimensions, or neurons, you can store at most N independent features orthogonally. Think of a cabinet with ten drawers. Each drawer holds one item. When you add the eleventh, there is nowhere left to put it.

But a model is not an organizer. It is compression software. It does not care how you categorize things. It cares about occupied space.

When features are sparse—when only a few are active at any given moment—the model can use superposition to store more than N features. This is less like organizing folders and more like bit-level compression on a high-density drive. Multiple features that rarely co-occur are encoded along non-orthogonal directions. Interference appears only when they activate together. As long as the data is not read at the same time, multiple files can occupy the same sector.

That compression has a trade-off. When two features share the same space, they create interference. The model relies on nonlinear activation functions such as ReLU to filter that interference. Otherwise, the output becomes muddled.

Superposition is not a bug. It is an efficient compression strategy for a capacity-constrained model. It lets a model use m dimensions to represent more than m sparse features.

The Core Mechanism: A Phase Change From Linear to Nonlinear

To understand how superposition happens, we need to look at how the model learns. Researchers used a simple toy model: the input layer receives features, the hidden layer transforms them through a weight matrix, and the output layer produces predictions.

The key is the introduction of the ReLU activation function.

In a purely linear bottleneck autoencoder, if the loss function is squared reconstruction error, the optimal solution can only store principal components. It cannot separate individual features. But once ReLU is added (x′ = ReLU(WTh+b)), a phase change occurs.

The research found that, across particular toy-model settings, loss functions, and sparsity/importance sweeps, the optimal solution can switch discretely among three configurations:

  1. No learning: The feature is not important enough, so it is ignored.
  2. Superposition learning: Multiple features are compressed into one or several neurons and separated through geometric structure.
  3. Dedicated dimensions: For extremely important or dense features, the model assigns dedicated neurons.

This transition is not gradual. It is sharp. Like water freezing at zero degrees, once sparsity reaches a critical point, the model can suddenly switch into superposition mode.

ReLU is the switch. It lets the model store features in a non-orthogonal space. Through a nonlinear filter, it truncates part of the negative interference. It does not eliminate interference, but it gives the model enough room to operate within limited dimensions.

The Geometry: Why Regular Polytopes?

This is the most surprising part. When multiple features are compressed into a low-dimensional space, they are not distributed randomly. They organize into specific geometric structures.

Researchers found that these structures correspond to the geometry of uniform polytopes. For example:

  • Two-dimensional space: Two features form antipodal pairs, pointing in opposite directions; pentagons involve more complex two-dimensional projections.
  • Three-dimensional space: Four features may form a tetrahedron; square antiprisms appear in projections from higher dimensions.

This geometric arrangement may look elegant, but it is actually dangerous. It is like parking cars side by side in a narrow alley: orderly at first glance, yet a small disturbance can propagate through the entire row. The more compact the geometry, the greater the cost of interference.

What Matters Most: Energy Jumps and Adversarial Vulnerability

Beyond geometry, the research reveals two phenomena with meaningful practical implications.

First, discrete energy level jumps. During parameter sweeps or training, a feature’s dimensional allocation does not change continuously. It jumps. The model switches between geometric configurations, like electrons transitioning between atomic orbitals. This helps explain why feature dimensions can shift discretely during certain stages of training.

Second, adversarial vulnerability. The research found that when a model enters a superposition state, its resistance to adversarial examples can decrease sharply. In the reported setting, vulnerability increased by more than 3x after superposition formed. Superposition introduces feature interference, allowing small input perturbations to be amplified through the geometric structure and produce incorrect outputs.

Models in superposition are more sensitive to adversarial attacks, with vulnerability increasing by more than 3x. This means high compression rates can come with a security trade-off, one worth considering when designing highly reliable systems.

Note that this >3x result comes from a specific toy model, with an attack limit of 0.1 times the average input L2 norm. Extending it to real-world models still requires careful evaluation.

Your Call: When Should You Care About Superposition?

Understanding superposition is not about fearing it. It is about using or mitigating it more effectively.

Good-fit situations:

  • Sparse and diverse features: If your task involves many concepts that rarely appear together, such as natural language processing, superposition is part of how models operate efficiently.
  • Moderate interpretability needs: Superposition makes individual neurons harder to interpret, but geometric structures and activation patterns can still reveal part of the model’s behavior.

Less suitable situations:

  • Highly correlated and dense features: If features frequently occur together, interference from superposition can lead to substantial output errors. In this setting, the model may favor dedicated dimensions or principal component analysis (PCA). In practice, tools such as SAEs can help “unpack” the representation rather than attempting to eliminate superposition itself.
  • High security requirements: As noted above, models in superposition can be more vulnerable to adversarial attacks. In domains such as healthcare or finance, additional defenses or a different architecture may be appropriate.

This is a trade-off. Superposition is a natural tension between neural-network efficiency and interpretability.

Next: From Black Box to X-Ray Lens

Anthropic’s research provides a solid foundation for mechanistic interpretability. It shows that neural networks are not internally chaotic. They are full of mathematical regularities.

For engineers, this opens several directions worth exploring:

  1. Measure superposition: Introduce sparse features into your model and observe the geometric structure of hidden-layer activations.
  2. Evaluate interference: Measure output interference across different feature combinations to identify higher-risk feature pairs.
  3. Explore SAEs: Apply sparse autoencoders to your model and see whether they improve interpretability without a significant performance trade-off.

If your model performs well on sparse data and interpretability needs are low, letting it remain in superposition can be a reasonable choice. If you need a high security standard, consider adding an SAE as a “decompressor.” This is not about right or wrong. It is about choosing for the context.

So the next time you see a confusing activation vector, do not rush to debug it. Ask first: is this noise, or compression? If it is compression, what trade-off are you prepared to make for that efficiency?

Sources