{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convolutional neural networks from scratch\n",
"\n",
"Now let's take a look at *convolutional neural networks* (CNNs), the models people really use for classifying images. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from __future__ import print_function\n",
"import mxnet as mx\n",
"import numpy as np\n",
"from mxnet import nd, autograd, gluon\n",
"ctx = mx.cpu()\n",
"# ctx = mx.gpu()\n",
"mx.random.seed(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## MNIST data (last one, we promise!)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"batch_size = 64\n",
"num_inputs = 784\n",
"num_outputs = 10\n",
"def transform(data, label):\n",
" return nd.transpose(data.astype(np.float32), (2,0,1))/255, label.astype(np.float32)\n",
"train_data = gluon.data.DataLoader(gluon.data.vision.MNIST(train=True, transform=transform),\n",
" batch_size, shuffle=True)\n",
"test_data = gluon.data.DataLoader(gluon.data.vision.MNIST(train=False, transform=transform),\n",
" batch_size, shuffle=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convolutional neural networks (CNNs)\n",
"\n",
"In the [previous example](5a-mlp-scratch.ipynb), we connected the nodes of our neural networks in what seems like the simplest possible way. Every node in each layer was connected to every node in the subsequent layers. \n",
"\n",
"![](https://github.com/zackchase/mxnet-the-straight-dope/blob/master/img/multilayer-perceptron.png?raw=true)\n",
"\n",
"This can require a lot of parameters! If our input were a 256x256 color image (still quite small for a photograph), and our network had 1,000 nodes in the first hidden layer, then our first weight matrix would require (256x256x3)x1000 parameters. That's nearly 200 million. Moreover the hidden layer would ignore all the spatial structure in the input image even though we know the local structure represents a powerful source of prior knowledge. \n",
"\n",
"Convolutional neural networks incorporate convolutional layers. These layers associate each of their nodes with a small window, called a *receptive field*, in the previous layer, instead of connecting to the full layer. This allows us to first learn local features via transformations that are applied in the same way for the top right corner as for the bottom left. Then we collect all this local information to predict global qualities of the image (like whether or not it depicts a dog). \n",
"\n",
"![](http://cs231n.github.io/assets/cnn/depthcol.jpeg)\n",
"(Image credit: Stanford cs231n http://cs231n.github.io/assets/cnn/depthcol.jpeg)\n",
"\n",
"In short, there are two new concepts you need to grok here. First, we'll be introducting *convolutional* layers. Second, we'll be interleaving them with *pooling* layers. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Parameters\n",
"\n",
"Each node in a convolutional layer is associated with a 3D block (height x width x channel) in the input tensor. Moreover, the convolutional layer itself has multiple output channels. So the layer is parameterized by a 4 dimensional weight tensor, commonly called a *convolutional kernel*. \n",
"\n",
"The output tensor is produced by sliding the kernel across the input image skipping locations according to a pre-defined *stride* (but we'll just assume that to be 1 in this tutorial). Let's initialize some such kernels from scratch."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#######################\n",
"# Set the scale for weight initialization and choose \n",
"# the number of hidden units in the fully-connected layer \n",
"####################### \n",
"weight_scale = .01\n",
"num_fc = 128\n",
"num_filter_conv_layer1 = 20\n",
"num_filter_conv_layer2 = 50\n",
"\n",
"W1 = nd.random_normal(shape=(num_filter_conv_layer1, 1, 3,3), scale=weight_scale, ctx=ctx) \n",
"b1 = nd.random_normal(shape=num_filter_conv_layer1, scale=weight_scale, ctx=ctx)\n",
"\n",
"W2 = nd.random_normal(shape=(num_filter_conv_layer2, num_filter_conv_layer1, 5, 5),\n",
" scale=weight_scale, ctx=ctx)\n",
"b2 = nd.random_normal(shape=num_filter_conv_layer2, scale=weight_scale, ctx=ctx)\n",
"\n",
"W3 = nd.random_normal(shape=(800, num_fc), scale=weight_scale, ctx=ctx)\n",
"b3 = nd.random_normal(shape=num_fc, scale=weight_scale, ctx=ctx)\n",
"\n",
"W4 = nd.random_normal(shape=(num_fc, num_outputs), scale=weight_scale, ctx=ctx)\n",
"b4 = nd.random_normal(shape=num_outputs, scale=weight_scale, ctx=ctx)\n",
"\n",
"params = [W1, b1, W2, b2, W3, b3, W4, b4]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And assign space for gradients"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"for param in params:\n",
" param.attach_grad()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convolving with MXNet's NDArrray\n",
"\n",
"To write a convolution when using *raw MXNet*, we use the function ``nd.Convolution()``. This function takes a few important arguments: inputs (``data``), a 4D weight matrix (``weight``), a bias (``bias``), the shape of the kernel (``kernel``), and a number of filters (``num_filter``)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for data, _ in train_data:\n",
" data = data.as_in_context(ctx)\n",
" break\n",
"conv = nd.Convolution(data=data, weight=W1, bias=b1, kernel=(3,3), num_filter=num_filter_conv_layer1)\n",
"print(conv.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note the shape. The number of examples (64) remains unchanged. The number of channels (also called *filters*) has increased to 20. And because the (3,3) kernel can only be applied in 26 different heights and widths (without the kernel busting over the image border), our output is 26,26. There are some weird padding tricks we can use when we want the input and output to have the same height and width dimensions, but we won't get into that now."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Average pooling\n",
"\n",
"The other new component of this model is the pooling layer. Pooling gives us a way to downsample in the spatial dimensions. Early convnets typically used average pooling, but max pooling tends to give better results. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pool = nd.Pooling(data=conv, pool_type=\"max\", kernel=(2,2), stride=(2,2))\n",
"print(pool.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the batch and channel components of the shape are unchanged but that the height and width have been downsampled from (26,26) to (13,13)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Activation function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def relu(X):\n",
" return nd.maximum(X,nd.zeros_like(X))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Softmax output"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def softmax(y_linear):\n",
" exp = nd.exp(y_linear-nd.max(y_linear))\n",
" partition = nd.sum(exp, axis=0, exclude=True).reshape((-1,1))\n",
" return exp / partition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Softmax cross-entropy loss\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def softmax_cross_entropy(yhat_linear, y):\n",
" return - nd.nansum(y * nd.log_softmax(yhat_linear), axis=0, exclude=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define the model\n",
"\n",
"Now we're ready to define our model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def net(X, debug=False):\n",
" ########################\n",
" # Define the computation of the first convolutional layer\n",
" ########################\n",
" h1_conv = nd.Convolution(data=X, weight=W1, bias=b1, kernel=(3,3),\n",
" num_filter=num_filter_conv_layer1)\n",
" h1_activation = relu(h1_conv)\n",
" h1 = nd.Pooling(data=h1_activation, pool_type=\"avg\", kernel=(2,2), stride=(2,2))\n",
" if debug:\n",
" print(\"h1 shape: %s\" % (np.array(h1.shape)))\n",
" \n",
" ########################\n",
" # Define the computation of the second convolutional layer\n",
" ########################\n",
" h2_conv = nd.Convolution(data=h1, weight=W2, bias=b2, kernel=(5,5),\n",
" num_filter=num_filter_conv_layer2)\n",
" h2_activation = relu(h2_conv)\n",
" h2 = nd.Pooling(data=h2_activation, pool_type=\"avg\", kernel=(2,2), stride=(2,2))\n",
" if debug:\n",
" print(\"h2 shape: %s\" % (np.array(h2.shape)))\n",
" \n",
" ########################\n",
" # Flattening h2 so that we can feed it into a fully-connected layer\n",
" ########################\n",
" h2 = nd.flatten(h2)\n",
" if debug:\n",
" print(\"Flat h2 shape: %s\" % (np.array(h2.shape)))\n",
" \n",
" ########################\n",
" # Define the computation of the third (fully-connected) layer\n",
" ########################\n",
" h3_linear = nd.dot(h2, W3) + b3\n",
" h3 = relu(h3_linear)\n",
" if debug:\n",
" print(\"h3 shape: %s\" % (np.array(h3.shape)))\n",
" \n",
" ########################\n",
" # Define the computation of the output layer\n",
" ########################\n",
" yhat_linear = nd.dot(h3, W4) + b4\n",
" if debug:\n",
" print(\"yhat_linear shape: %s\" % (np.array(yhat_linear.shape)))\n",
" \n",
" return yhat_linear\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Test run\n",
"\n",
"We can now print out the shapes of the activations at each layer by using the debug flag."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"output = net(data, debug=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Optimizer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def SGD(params, lr): \n",
" for param in params:\n",
" param[:] = param - lr * param.grad"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluation metric"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def evaluate_accuracy(data_iterator, net):\n",
" numerator = 0.\n",
" denominator = 0.\n",
" for i, (data, label) in enumerate(data_iterator):\n",
" data = data.as_in_context(ctx)\n",
" label = label.as_in_context(ctx)\n",
" label_one_hot = nd.one_hot(label, 10)\n",
" output = net(data)\n",
" predictions = nd.argmax(output, axis=1)\n",
" numerator += nd.sum(predictions == label)\n",
" denominator += data.shape[0]\n",
" return (numerator / denominator).asscalar()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The training loop"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"epochs = 1\n",
"learning_rate = .01\n",
"smoothing_constant = .01\n",
"\n",
"for e in range(epochs):\n",
" for i, (data, label) in enumerate(train_data):\n",
" data = data.as_in_context(ctx)\n",
" label = label.as_in_context(ctx)\n",
" label_one_hot = nd.one_hot(label, num_outputs)\n",
" with autograd.record():\n",
" output = net(data)\n",
" loss = softmax_cross_entropy(output, label_one_hot)\n",
" loss.backward()\n",
" SGD(params, learning_rate)\n",
" \n",
" ##########################\n",
" # Keep a moving average of the losses\n",
" ##########################\n",
" curr_loss = nd.mean(loss).asscalar()\n",
" moving_loss = (curr_loss if ((i == 0) and (e == 0)) \n",
" else (1 - smoothing_constant) * moving_loss + (smoothing_constant) * curr_loss)\n",
" \n",
" \n",
" test_accuracy = evaluate_accuracy(test_data, net)\n",
" train_accuracy = evaluate_accuracy(train_data, net)\n",
" print(\"Epoch %s. Loss: %s, Train_acc %s, Test_acc %s\" % (e, moving_loss, train_accuracy, test_accuracy))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"Contained in this example are nearly all the important ideas you'll need to start attacking problems in computer vision. While state-of-the-art vision systems incorporate a few more bells and whistles, they're all built on this foundation. Believe it or not, if you knew just the content in this tutorial 5 years ago, you could probably have sold a startup to a Fortune 500 company for millions of dollars. Fortunately (or unfortunately?), the world has gotten marginally more sophisticated, so we'll have to come up with some more sophisticated tutorials to follow."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next\n",
"[Convolutional neural networks with gluon](../chapter04_convolutional-neural-networks/cnn-gluon.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"For whinges or inquiries, [open an issue on GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}