MNIST

Directions

Draw a single digit number (0-9) in the box on the left. A convolutional neural network will then classify your drawing and its prediction will be highlighted in blue. Your drawing must be cropped and scaled down to 28x28 pixels before being fed into the network. This preprocessing is shown in the smaller image below the prediction.

About

This is a convolutional neural network that I wrote from scratch, without using any machine learning library like TensorFlow.

It was trained with the MNIST database, which contains 60,000 images of handwritten digits. The first 50,000 images were used to train, and the next 10,000 were used to test accuracy. The training set was expanded to 150,000 images by rotating each image 15 degrees in both directions. This network correctly classified 9,959 of the 10,000 test images.

The optimizer I created used stochastic gradient decent with the following parameters:

• batch size 10
• 0.001 learning rate, 0.002 decay, and 0.9 momentum.
• 0.001 l2 regularization

These are the layers that make up the network. You can learn more about them in the resources below.

Resources

3 Blue 1 Brown

The YouTube channel 3 Blue 1 Brown has an excellent series on neural networks. The author is very good at helping you visualize the way neural networks function and learn. It is a four-part series that begins as a simple introduction and ends by walking you through the complex math involved in backpropagation.

Here are links to each part in the series:

1. But what *is* a Neural Network?
2. Gradient descent, how neural networks learn.
3. What is backpropagation really doing?
4. Backpropagation calculus.

Michael Nielsen

One of the best resources to help you program your own neural network from scratch is Michael Nielsen's e-book titled Neural Networks and Deep Learning. He does an excellent job explaining the how's and why's of neural networks. More importantly, he provides all his source code, so you can easily load the MNIST data sets and debug his network alongside your own. If you work through the first 3 chapters of this book you will be able to create your own fully connected, feed forward network that can classify MNIST data with over 98.0% accuracy!

Python

Most machine learning resources use Python. This includes Michael Nielsen's book above. You can download the latest version at https://www.python.org/downloads/. If you are new to Python you will want to visit The Python Tutorial.

Stanford CS

The Stanford CS class CS231n: Convolution Neural Networks for Visual Recognition is a great resource to supplement Michael Neilson's book. Working through the materials on this site will help you add convolutional, max pooling, and dropout layers to your network to achieve over 99.0% accuracy!

This site also has a very helpful Python NumPy Tutorial.

Mind

If you want to start with something simpler you may want to read Mind: How to Build a Neural Network (Part One). Instead of working with complex MNIST data, this article walks you through training a neural network to function as an XOR operation using only two bits as input. It works through some of the complex math involved in forward and backpropagation and provides specific weight and bias initialization values for you to follow along with in your own network.