Zoom In: An Introduction to Circuits
By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks.
Authors
Affiliations
Published
March 10, 2020
DOI
This article is part of the Circuits thread, an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.
Circuits Thread An Overview of Early Vision in InceptionV1
Many important transition points in the history of science have been moments when science “zoomed in.” At these points, we develop a visualization or tool that allows us to see the world in a new level of detail, and a new field of science develops to study the world through this lens.
For example, microscopes let us see cells, leading to cellular biology. Science zoomed in. Several techniques including x-ray crystallography let us see DNA, leading to the molecular revolution. Science zoomed in. Atomic theory. Subatomic particles. Neuroscience. Science zoomed in.
These transitions weren’t just a change in precision: they were qualitative changes in what the objects of scientific inquiry are. For example, cellular biology isn’t just more careful zoology. It’s a new kind of inquiry that dramatically shifts what we can understand.
The famous examples of this phenomenon happened at a very large scale, but it can also be the more modest shift of a small research community realizing they can now study their topic in a finer grained level of detail.
Hooke’s Micrographia
-
Micrographia: or Some Physiological Descriptions of Minute Bodies Made by Magnifying Glasses. With Observations and Inquiries Thereupon [link]
R. Hooke.
The Royal Society. 1666.
[1]
revealed a rich microscopic world as seen through a microscope, including the initial discovery of cells.
Images from the National Library of Wales.
Just as the early microscope hinted at a new world of cells and microorganisms, visualizations of artificial neural networks have revealed tantalizing hints and glimpses of a rich inner world within our models (e.g.
-
Visualizing and understanding recurrent networks [PDF]
A. Karpathy, J. Johnson, L. Fei-Fei.
arXiv preprint arXiv:1506.02078. 2015.
-
Visualizing higher-layer features of a deep network [PDF]
D. Erhan, Y. Bengio, A. Courville, P. Vincent.
University of Montreal, Vol 1341, pp. 3. 2009.
-
Feature Visualization [link]
C. Olah, A. Mordvintsev, L. Schubert.
Distill. 2017.
-
Deep inside convolutional networks: Visualising image classification models and saliency maps [PDF]
K. Simonyan, A. Vedaldi, A. Zisserman.
arXiv preprint arXiv:1312.6034. 2013.
-
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images [PDF]
A. Nguyen, J. Yosinski, J. Clune.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427—436. 2015.
-
Inceptionism: Going deeper into neural networks [HTML]
A. Mordvintsev, C. Olah, M. Tyka.
Google Research Blog. 2015.
-
Plug & play generative networks: Conditional iterative generation of images in latent space [PDF]
A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, J. Yosinski.
arXiv preprint arXiv:1612.00005. 2016.
-
Visualizing and understanding convolutional networks [PDF]
M.D. Zeiler, R. Fergus.
European conference on computer vision, pp. 818—833. 2014.
-
Interpretable Explanations of Black Boxes by Meaningful Perturbation [PDF]
R. Fong, A. Vedaldi.
arXiv preprint arXiv:1704.03296. 2017.
-
PatternNet and PatternLRP—Improving the interpretability of neural networks [PDF]
P. Kindermans, K.T. Schutt, M. Alber, K. Muller, S. Dahne.
arXiv preprint arXiv:1705.05598. 2017.
-
Visualizing and Measuring the Geometry of BERT [PDF]
E. Reif, A. Yuan, M. Wattenberg, F.B. Viegas, A. Coenen, A. Pearce, B. Kim.
Advances in Neural Information Processing Systems, pp. 8592—8600. 2019.
-
Activation atlas [link]
S. Carter, Z. Armstrong, L. Schubert, I. Johnson, C. Olah.
Distill, Vol 4(3), pp. e15. 2019.
[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
). This has led us to wonder: Is it possible that deep learning is at a similar, albeit more modest, transition point?
Most work on interpretability aims to give simple explanations of an entire neural network’s behavior. But what if we instead take an approach inspired by neuroscience or cellular biology — an approach of zooming in? What if we treated individual neurons, even individual weights, as being worthy of serious investigation? What if we were willing to spend thousands of hours tracing through every neuron and its connections? What kind of picture of neural networks would emerge?
In contrast to the typical picture of neural networks as a black box, we’ve been surprised how approachable the network is on this scale. Not only do neurons seem understandable (even ones that initially seemed inscrutable), but the “circuits” of connections between them seem to be meaningful algorithms corresponding to facts about the world. You can watch a circle detector be assembled from curves. You can see a dog head be assembled from eyes, snout, fur and tongue. You can observe how a car is composed from wheels and windows. You can even find circuits implementing simple logic: cases where the network implements AND, OR or XOR over high-level visual features.
Over the last few years, we’ve seen many incredible visualizations
-
Visualizing and understanding recurrent networks [PDF]
A. Karpathy, J. Johnson, L. Fei-Fei.
arXiv preprint arXiv:1506.02078. 2015.
-
Visualizing higher-layer features of a deep network [PDF]
D. Erhan, Y. Bengio, A. Courville, P. Vincent.
University of Montreal, Vol 1341, pp. 3. 2009.
-
Feature Visualization [link]
C. Olah, A. Mordvintsev, L. Schubert.
Distill. 2017.
-
Deep inside convolutional networks: Visualising image classification models and saliency maps [PDF]
K. Simonyan, A. Vedaldi, A. Zisserman.
(Excerpt: introduction and the opening of the three speculative claims; full text at source URL. DOI: 10.23915/distill.00024.001)