Visualizing Convolution Kernels

I very rarely see any sort of inspection being done on the convolutional kernels of a CNN. In part this is because the parameters themselves are far more difficult to interpret than the outputs of a network (even intermediate outputs a.k.a the network activations). This difficulty of interpretation is worst for kernels with a small spatial footprint and unfortunately 3x3 kernels are the most performant and popular choice. Trying to understand the structure of a 3x3 convolution kernel by looking at all of the possible 3x3 spatial slices is somewhat like trying to guess what an full image looks like from being shown all the 3x3 chunks of it in random order.

Despite the difficulties I think good kernel visualizations are a worthwile pursuit. Good visualization techniques can be powerful diagnostics and the better the visualizations of our models the more powerful and robust we can make them. As a motivational carrot here is a teaser plot of a visualization of a simple network which we generate in this post.

Work on visualizing neural networks has mostly dealt with how to visualize the activations within the network when we pass in test images (or at least the work I am familiar with). Two common approaches are to either look at heat maps of the activations of particular layers/channels, or to look directly at the images which induce large magnitude activations in those same layers/channels. A particularly nice tool called lucid is particularly good for making visualizations of the second sort.

In this post I don't seek to visualize activations resulting from a set of parameters per-se (though that is helpful). My main goal is to directly visualize the parameter values of the network themselves. To put this distinction in a familiar context consider a simple linear model $y=Ax$. The values in the transform matrix A are the parameters of the model, the values $y$ are the output "activations" of the model and the vector $x$ is the input example. The technique of visualizing the activations of a neural network is analogous to plotting the $y$ resulting from a given $x$. The technique of finding input examples which maximize layer activations is analogous to plotting up the eigenvector of $A$ of maximum magnitude eigenvalue. Attempting to directly visualize the convolution kernel parameters would be analogous to plotting up the rows of $A$. All of these are very useful and complementary tools for understanding a linear model. Of course the analogy is somewhat imperfect since neural networks can be very non-linear but I think the point is valid. Most analysts wouldn't want to give up the freedom to inspect the coefficients of a linear model so why should we give up on inspecting the parameters of a neural network simply because they are harder to interpret?

An Example Network¶

A post about visualizations would be a sad post indeed if we didn't actually visualize anything in it. Lets train a small neural network for predicting the labels on the cifar-10 dataset to play around with.

Lets save the kernel parameters for later analysis.

Stacked Heatmaps¶

It seems to me that everyone's first impulse when asked to think of a way to visualize a convolution kernel is simply to plot 2D slices of it as heatmaps. Using this technique we can visualize the values in the entire 4D tensor by making one such heatmap for each possible slice through the remaining 2 dimensions. This is a very limted visualization strategy since it makes interpreting the cross channel information very difficult but its straight forward and intuitive nature makes up for its failings.

Negative entries in the convolution kernel are qualitatively different than positive entries and so it is a good idea to use a divergent color map centered on zero for the heatmaps. I'm particularly fond of using a blue-black-orange divergent color map.

These visualizations are certainly fun to look at but gleaning meaningful information from them is difficult. Especially as we move deeper in to the network the visualization comes to look more and more like random noise. There is no meaningful sense of adjacency for input or output channels. The fact that the channel of index 0 is next to the channel at index 1 is pure coincidence. Any random permutation of input or output channels would be just as valid. So although the adjacency in each displayed 3x3 slice is meaningful the larger scale structure is not.

As we move deeper into the network the important information becomes more and more shifted towards the channel dimensions instead of the spatial dimensions and there is more and more information in the values of the kernels. The visualization above doesn't give much of a sense for how fundamentally different in scale and in type the sort of information present in the small initial kernels versus the later large kernels really is. Because we are rescaling each image to the same figure size we get to clearly see all the details for the first kernel and we at least assign around 1 pixel per parameter value for most of the kernels. But the last layer isn't able to be properly embedded in such a small image at 300 dpi (the matplotlib default). The apparent checkerboard effect you can see in the image for conv3 above is just an aliasing artifact, not actually a true structural effect.

We can fix the aliasing problem and give a sense for the true relative sizes of the kernels by scaling each image size to be proportional to the number of input/output channels in each (plus a term for kernel width and height). Lets also get rid of the colorbar and turn the matplotlib title into a print statement so that they don't mess with the relative scaling of the axes. This way each parameter value gets roughly the same amount of screen real estate no matter which layer it belongs to.

Looking at these last set of visualizations I think nicely illustrates the sheer scale of the problem. A simple grid of heatmaps works mildly well for visualizing the input layer of a network or even the first few layers, just as long as the number of input/output channels is relatively small. But for deeper network layers there is just too much information stored in the channelwise dimensions of the kernels to make this a really effective strategy.

Playing Games with Parameter Ordering¶

Because we are choosing to preserve the adjacency of the pixels along the spatial dimensions in the above visualizations we are giving quite a lot of emphasis to the spatial structure of the kernels. As the spatial kernel structure becomes more and more dilute relative to the cross channel structure of the kernels of course the visualizations become less understandable.

We can try to emphasize the channel information instead by picking slices through the kernel along the input and output channel axes instead of the spatial axes and then stacking these slices by pixel order.

Since the channel ordering is effectively random it should be no real surprise that the effect of preserving the meaningless channel adjacency at the price of the spatial adjacency results in something much more chaotic looking than before.

There is likely no "correct" way to pick a channel ordering but anything is better than completely random right? Well I am not going to junk up the post with images of all the different things I tried. I tried agglomerative clustering like in my post about visualizing correlation heatmaps, I tried ordering by variance of the kernels over input and output channels, I tried ordering by central magnitude, I tried ordering by mean magnitude, I tried ordering by mean value, I tried ordering by a specialized heuristic search to find the best chain of rows which look something like a progressive sequence. All of these attempts resulted in things which looked essentially like random garbage. The progressive sequence search seemed most promising but the implementation is problematic and potentially buggy, it may turn into a post of its own in the future.

But for the current task of making good visualizations of convolution kernels I think the moral is clear. Without the context which the network activations, data, and parameters provide the values in the kernels just look essentially like the random noise.

In fact...

They look alot like the very same random noise with which they were originally initialized! There are an uncountably vast number of local optima for our chosen loss function and data set pairing. It makes sense that the local optimum we would fall into should be "close" to our starting parameters. So it is perhaps not surprising that since we started with kernels that look like random Gaussian noise we get parameters out that look very similar.

This effect is somewhat subtle and although it is easy to do a by eye comparison between the initial and trained values for the very first kernel it gets way too busy to do for any of the other kernels. Instead lets do a distributional analysis.

As you can see, the residuals between the initial parameters and the final trained parameters is much better than random (since the residual distribution is narrower than the trained parameter value distribution itself).

The effect is only really strong for the input layer and becomes progressively weaker as we move to deeper layers but persists never the less. We can also see that there is a noticeable shift towards negative values in the trained parameters, something you might also have gleaned from the dominance of blue in the stacked heatmap visualizations above.

Going Deeper¶

The stacked heatmap visualization we worked with here is somewhat useful (and pretty) as it stands. But there will be little to no consistency between visualizations of different networks, or even the same network trained with different initializations. In order to have any hope of making the visualizations truly meaningful we need somehow to incorporate some sort of standardizing context.

My first impulse is to apply PCA to the activations being input into each convolution layer and to then project the kernels onto the principal vectors. However as I started to implement that idea I realized that the non-linearity introduced by the relu activation function severely limits the accuracy of a PCA expansion of the activations. Accounting for this complication and doing a really proper job of modeling the activation model activation distribution is a deep enough rabbit hole that it makes sense to break it out into its own blog post or perhaps even a series of posts.

Even with a good activation distributional model in hand the challenge of meaningfully organizing the input/output channel information remains a fundamentally tough problem. The good news is that once we have a distributional model over patches of input activations then we can collapse the first three dimensions of the kernel against artfully chosen samples from the activation distribution which will turn the complex and unfamiliar problem of considering a 4D convolution kernel into the much more familiar problem of considering a linear transform matrix $x_2=x_1 W + b$. This has the additional benefit that we can then apply exactly the same techniques to visualize both convolution kernels and dense network kernels.