(It is a backup of the post from my wordpress https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/, and the Chinese version is published on logdown too http://phunter.logdown.com/posts/314562)
I am going to have a series of blogs about implementing deep learning models and algorithms with MXnet. The topic list covers MNIST, LSTM/RNN, image recognition, neural artstyle image generation etc. Everything here is about programing deep learning (a.k.a. deep learning for hackers), instead of theoritical tutorials, so basic knowledge of machine learning and neural network is a prerequisite. I assume readers know how neural network works and what
backpropagation is. If difficulties, please review Andew Ng's coursera class Week 4https://www.coursera.org/learn/machine-learning.
Surely, this blog doesn't show everything about deep learning. It is very important to understand the fundamental deep learning knowledge. For readers who want to know in-depth theoritical deep learning knowledge, please read some good tutorials, for example, http://deeplearning.net/reading-list/tutorials/.
MXnet: lightweight, distributed, portable deep learning toolkit
MXnet is a deep learning toolkit written in C++11, and it comes with DMLC (Distributed (Deep) Machine Learning Common
http://dmlc.ml/). You might have known MXnet's famous DMLC-sibling
xgboost https://github.com/dmlc/xgboost, a parallel gradient boosting decision tree which dominates most Kaggle competitions and is generally used in many projects.
MXnet is very lightweight, dynamic, portable, easy to distribute, memory efficient, and one of the coolest features is, it can run on portable devices (e.g. image recognition on your Android phone ) MXnet also has clear design plus clean C++11 code, let go star and fork it on github: https://github.com/dmlc/mxnet
Recently MXnet has received much attention in multiple conferences and blogs for its unique features of speed and efficient memory usage. Professionals are comparing MXnet with Caffe, Torch7 and Google's TensorFlow. These benchmarks show that MXnet is a new raising star. Go check this recent tweet from Quora's Xavier Amatriain: https://twitter.com/xamat/status/665222179668168704
Install MXnet with GPU
The installation can be done on physical machines with nVidia CUDA GPUs or cloud instance, for example AWS GPU instance
g2.8xlarge. The following steps mostly come from the official installation guide http://mxnt.ml/en/latest/build.html#building-on-linux, with some CUDA modification.
Please note: for installing CUDA on AWS from scratch, some additional steps are needed for updating
linux-image-extra-virtual and disabling
nouveau, for more details, please refer to Caffe's guide: https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-Ubuntu,-CUDA-7,-cuDNN)(
MXnet only needs minimal dependency: gcc, BLAS, and OpenCV (optional), that is it. One can install
git just in case it hasn't been installed.
Just another reminder that
--recursive is needed: MXnet depends on DMLC common packages
--recursive can clone all necessary ones. Please don't compile now, and we need to install CUDA firstly.
CUDA installation here is universal for other deep learning packages. Please go to https://developer.nvidia.com/cuda-downloads for selecting the CUDA installation for the corresponding system. For example, installing CUDA for Ubuntu 14.04 should looks like this, and
deb(network) is suggested for fastest downloading from the closest Ubuntu source.
Or, here it is the command-line-only solution：
If everything goes well, plese check the video card status by
nvidia-smi, and it should look like this：
CPU info may vary, and I am using a GTX 960 4GB (approximately 200$ now). MXnet has very efficient memory usage, and 4GB is good for most of the problems.
Optional: CuDNN: Mxnet supports
cuDNN is nVidia deep learning toolkit which optimizes operations like convolution, poolings etc, for better speed and memory. Usually it can speed up MXnet by 40% to 50%. If interested, please go apply for the developer program here https://developer.nvidia.com/cudnn, and install
cuDNN by the official instruction when approved,
Compile MXnet with CUDA support
MXnet needs to turn on CUDA support in the configuration. Please find
mxnet/make/, copy to
mxnet/, and edit these three lines:
where the second line is for CUDA installation path. The path usually is
/usr/local/cuda-7.5. If readers prefer other
BLAS implementations. e.g.
Atlas, please change
atlas and add the blas path to
We can compile MXnet with CUDA (
-j4 for multi-thread compiling):
One more reminder that, if one has non-CUDA video cards, for example Intel Iris or AMD R9, or there is not video card, please change
0. MXnet is dynamic for switching between CPU and GPU: instead of GPU version, one can compile multi-theading CPU version by setting
USE_OPENMP = 1 or leave it to 0 so
BLAS can take care of multi-threading, either way is fine with MXnet.
Install Python support
MXnet natively supports Python, one can simply do:
Python 2.7 is suggested while Python 3.4 is also supported. One might need
numpy if not yet installed. I personally suggest Python from
Let's run MNIST, a handwritten digit recognizer
Now we have a GPU-ready MXnet, let's have the first deep learning example: MNIST. MNIST is a handwritten digit dataset with 60,000 training samples and 10,000 testing samples, where each samples is a 28X28 greyscale picture of number, and the goal of MNIST is training a small machine learning model for recognizing hand-writing digits.
Let's run it:
train_mnist.py will download MNIST dataset for the first time, please be patient.
train_mnist.py by default uses CPU only. MXnet has easy swtich between CPU and GPU. Since we have GPU, let's turn it on by:
That is it.
--gpus 0 means using the first GPU. If one has multiple GPUs, for example 4 GPUs, one can set
--gpus 0,1,2,3 for using all of them. While running with GPU, the
nvidia-smi should look like this：
where one can see
python is using GPU. Since MNIST is not a heavy task, with MXnet efficient GPU meomory usage, GPU usage is about 30-40% while memory usage at 67MB。
When run with GPU for the first time, readers may see something like this：
It is because of the PATH of CUDA dynamic link lib, one can add this to
Or compile it to MXnet by adding in
MNIST code secrete revealed: design a simple MLP
train_mnist.py, there is an function
get_mlp(). It implements a multilayer perceptron (MLP). In MXnet, a MLP needs some definition, like this in the code:
Let's understand what is going on for this neural network. Samples in MNIST look like these:
- Each same (a digit) is a 28X28 pixel grey scale image, which can be represented as a vector of 28X28=784 float value where each value is the grey scale of the pixel.
- In MLP, each layer needs a layer structure. For example in the first layer,
fc1is a full connected layer
mx.symbol.FullyConnectedwhich takes input from
data. This first layer has 128 nodes, defined as
- Each layer also need an activation function
Activationto connect to the next layer, in other words, transferring values from the current layer to the next layer. In this example, the
Activationfunction for connecting layer
ReLu, which is short for
rectified linear unitor
Rectifier, a function as
ReLUis a very commonly used activation function in deep learning, mostly because it is easy to calculate and easy to converge in gradient decent. In addition, we choose
ReLUfor the MNIST problem because MNIST has sparse feature where most values are 0. For more information about
ReLU, please check wikipedia and other deep learning books or tutorials.
- The second layer
fc2is similar to the first layer
fc1: it takes the output from
fc1as input, and output to the third layer
fc3works similar to the previous layer, and the only difference is that, it is an output layer, which has 10 nodes where each node outputs the probability of being one of the 10 digits, thus
With this network structure, MXnet also needs the stucture of input. Since each sample is 28X28 grey scale, MXnet takes the grey scale value vector of 28X28=784 elements, and give a python iterator
get_iterator() for feeding data to the network defined above. The detailed code is in the example which is very clean, so I don't copy-n-paste here.
The final step is running the model. If readers know
scikit-learn, MXnet's python looks very familiar, right?
Congratulations! We can implement a MLP! It is the first step of deep learning, not so hard, right?
It is Q&A time now. Some of my dear readers may ask, "Do I need to design my MNIST recognizer or some other deep learning network exactly like what you did here?" "hey, I see another function
get_lenet() in the code, what is that?"
The answer to these two questions can be, most real life problems on deep learning are about designing the networks, and, no, you don't have to design your network exactly like what I did here. Designing the neural network is art, and each problem needs different networks. In the MNIST example code,
get_lenet() implements Yann Lecun's convolution network
LeNet for digit recognition, where each layer needs
Pooling where the kernel size and filter are needed, instead of
ReLU. FYI: the detailed explaination of super cool convolution network (ConvNet) can be found at Yann Lecun's tutorial: http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf . Another good reference can be "Understanding ConvNet" by Colah. I may later on write a blog post for explaining ConvNet, since convnet is my personal favorite.
I have a fun homework for my dear readers. Let's tune up some network parameters like the number of nodes and the activation function in
get_mlp(), and see how it helps the precision and accuracy. One can also try changing
num_epoch (number of iterations of learning from the data) and
learning_rate (the speed of gradient decent, a.k.a the rate of learning) for better speed and precision. Please leave your comment for your network structure and precision score. Kaggle also has a MNIST competition, one can also go compete with mxnet MNIST models, please mention it is MXnet model. The portal: https://www.kaggle.com/c/digit-recognizer
Thanks for reading this first blog of "Deep learning for hackers with MXnet". The following blogs will include some examples in MXnet, which may include RNN/LSTM for generating a Shakespeare script (well, looks like Shakespeare), generative models of simulating Van Gogh for painting a cat, etc. Some of these models are available now on MXnet github https://github.com/dmlc/mxnet/tree/master/example. Is MXnet cool? Go star and fork it on github https://github.com/dmlc/mxnet