Convolution and filters

Convolution, the basic operation of vision

A convolution slides a small grid of weights, the kernel, over the whole image. At each position, the output value is the sum of products between the kernel weights and the pixels it covers:

Nothing else: nine multiplications and one addition per pixel. The whole effect comes from the choice of weights.

Reading a kernel

• All-positive weights summing to 1: a weighted average. The image is smoothed (blur). Gaussian blur weighs the centre more and smooths without blocky artefacts.
• Weights summing to 0, positive on one side and negative on the other: a difference. The output is zero on a flat area and strong where intensity changes: an edge detector. Horizontal Sobel responds to vertical edges, and vice versa.
• Strong centre, negative neighbours: the image plus its own edges, hence a sharper image.

Try the test pattern: thin bars vanish under blur (high frequencies are attenuated), while Sobel only responds to their borders.

The link with convolutional networks

A convolutional neural network (CNN) applies exactly this operation, with one decisive difference: the kernel weights are not chosen, they are learned. The first layers of a CNN trained on images rediscover edge and texture detectors close to Sobel by themselves; later layers combine these responses into increasingly abstract patterns.

Two practical details, visible here:

• borders: the kernel overhangs the image at its rim. This demo repeats the border pixel; a CNN often adds zeros (padding);
• the "convolution" of deep learning libraries is a cross-correlation: the kernel is not flipped. That is also the case here. For a learned kernel, the distinction does not matter.