4 comments
I think certain classes of artificial neural networks are definitely more interpretable than others.<p>For example, a trillion-parameter transformer model built using many layers is likely <i>significantly</i> easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.<p>The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
Feels like I'm back in the distill.pub days!
65 minute read with plots for days all in order to change the semantic definition of "superposition" to "gating"
16,000 words into the essay: "What I mean by clean"<p>...and it still doesn't answer it. What is this actually about?
It means that modern neural networks are way more interpretable than older ones. We can understand how they work way more easily than earlier ones.
I think "clean" means something like each neuron encodes exactly one concept and it's possible to discover what the concept is.