Sparse features in small vision models

Most interpretability work on vision models has studied large ones, for good reasons: they are the models people deploy. We went the other way. A small model is cheap to probe exhaustively, which means you can ask not only “can we find interpretable features?” but “what fraction of the model can we explain at all?” The second question is the one we cared about.

Why small models

The encoder we studied has 90 million parameters and twelve layers.Note 11Ninety million parameters is small by current standards. It fits on a single laptop GPU, which mattered more to us than its benchmark scores. It is not state of the art, but it is competent: it supports Meridian 3’s image understanding at a fraction of the cost, and it was trained on the same data. If its internal representations are mostly legible, that tells us something about what larger encoders are likely to contain. If they are not, a small model is the cheapest place to find out why.

1. 1Ninety million parameters is small by current standards. It fits on a single laptop GPU, which mattered more to us than its benchmark scores.

Earlier work on small language models found that sparse dictionaries recover directions that each respond to one recognisable thing [1]. Vision has an extra difficulty: image patches overlap in content far more than tokens do, so a feature that looks clean on its top examples can turn out to be a mixture once you look further down the list.

The dictionary

We trained one sparse autoencoder per layer on 2.1 billion patch activations, with a dictionary sixteen times wider than the layer itself.Note 22We tuned the sparsity penalty so that an average patch activates about 38 features. Results were similar at 24 and 60. Our earlier probing work [2] suggested the middle layers would be the most interesting, and they were: layers six through eight hold most of the features that respond to parts of objects rather than raw texture or whole scenes.

1. 2We tuned the sparsity penalty so that an average patch activates about 38 features. Results were similar at 24 and 60.

What we found there will be familiar to anyone who has looked inside a vision model. There are curve detectors at many orientations, features for specular highlights, for the boundary between sky and anything else, for the repeated vertical strokes of printed text, and for fur. Many of them are strikingly clean.

Scoring a feature without squinting

“Strikingly clean” is the problem. People are very good at finding a pattern in twenty images, which makes looking at top-activating examples a poor test of whether a feature is understood. Recent critiques of feature descriptions make the same point [3].

So we scored descriptions by whether they discriminate. For each feature, one person wrote a short description from its top examples. A different person then saw two shuffled sets of patches, one of top activations and one random, and had to say which set the description fit.Note 33Raters saw 20 patches per set. Each description was tested by three raters, none of whom wrote it; we use the mean. A feature’s interpretability score is how much better than chance the second person did, scaled so that zero is chance and one is perfect.

1. 3Raters saw 20 patches per set. Each description was tested by three raters, none of whom wrote it; we use the mean.

Figure 1 shows the result for a random sample of features from layer seven. Most score well. The ones that score poorly are not concentrated at one end: very rare features and very common features both include failures, which rules out the easiest explanation, that we simply had too few examples of the rare ones.

The fifth we could not name

About a fifth of the dictionary falls below 0.4, the point below which raters could not reliably distinguish a feature’s top patches from random ones. We looked at these for a long time, and three things stood out.

First, many of them respond to position rather than content.Note 44We confirmed this by pasting the same object at different offsets on a blank background. The features followed the offset, not the object. They fire on patches at a fixed offset from an object’s edge, whatever the object is. A person looking at the patches sees nothing in common because the thing in common is not in the patch.

1. 4We confirmed this by pasting the same object at different offsets on a blank background. The features followed the offset, not the object.

Second, some seem to be parts of a larger feature split across several dictionary entries. Their activations are highly correlated with one or two neighbours, and descriptions of the group score well where descriptions of the individuals do not.

Third, a remainder, perhaps a third of the unnamed set, have no structure we could find with any method we tried. We do not know what they are.

What this changes for us

The share of nameable features rises through the middle layers and falls again near the output, as Figure 2 shows, which matches the intuition that late layers combine parts into abstractions people do not have words for. That is a reason for humility about descriptions of large models’ late layers, which are the ones most often studied.

For our own work, the practical change is that we now report an interpretability score alongside every feature we describe in a paper, and we report the fraction of the dictionary that falls below threshold. A claim that a model contains a feature for something is more useful when it comes with an estimate of how much of the model we could not explain at all.