Deep Learning in 7 lines of code
The essence of machine learning is recognizing patterns within data. This boils down to 3 things: data, software and math. What can be done in seven lines of code you ask? A lot.
The way to reduce a deep learning problem to a few lines of code is to use layers of abstraction, otherwise known as ‘frameworks’. Today we’ll use tensorflow and tflearn.
Abstraction is an essential property of software: the app you are using to view this piece is an abstraction layer above some operating system that knows how to read files, display images, etc. and this is an abstraction above lower level functions. Ultimately there is CPU-level code that moves bits — the ‘bare metal’.
Software frameworks are abstraction layers.
Our reduction is achieved by using tflearn, a layer above tensorflow, a layer above a Python. As always we’ll use iPython notebook as a tool to facilitate our work.
Let’s start at the beginning
In “How Neural Networks Work” we built a neural network in Python (no frameworks), and we showed how machine learning could ‘learn’ from patterns of data, using a ‘toy data’ example. Recall the ‘toy’ data is purposefully simple so that we can intuitively grok the patterns within it.
The notebook for this “zero abstraction” code is here. Each mathematical operation within the model is detailed.
Extending our model to use 2 hidden layers and Gradient Descent such as the one we built for analyzing text, we have ~80 lines of code, again sans frameworks. This is an example of “Deep Learning, the “depth” comes from the hidden layers.
The definition of our model is relatively straight-forward, albeit laborious, most of the code is applied to training:
This worked quite well — we then abstracted it using a framework.
Abstracting with Tensorflow
In “Tensorflow demystified” we built the same neural network, again we showed how machine learning could ‘learn’ from patterns of data.
This simplified our code (same underlying mathematical structures), for example the handling of Gradient Descent and loss function is conveniently reduced to 2 lines of code.
Our model’s definition is also simplified, the math and common functions (eg. sigmoid) are encapsulated for us in the framework.
You can imagine a complex neural network ‘flow’, like AlexNet, using tensorflow to simplify the definition and work for its mathematical ‘flow’.
Abstracting again
Still unsatisfied with the amount of code and complexity involved, we abstract again using tflearn, which describes itself as:
TFLearn: Deep learning library featuring a higher-level API for TensorFlow.
By “higher-level” they mean higher abstraction level, which is what we’re after. So we have our 7 lines of code for a multi-layer neural net.
This is magnificent — 5 lines of code to define our neural net structure (input +2 hidden +output +regression), 2 lines to train it.
Our notebook code is here.
Let’s go through this in detail, you’ll notice that the data and learning intent is identical to our earlier example.
Framework installation
Make sure you have tensorflow 1.0.x installed, the tflearn framework will not work with tensorflow prior to version 1.0
'1.0.1'It may be helpful (on Linux with pip) to use:
python -m pip install — upgrade tensorflow tflearn
Data
Next we setup our data, same toy data as our tensorflow example. The training data was explained in detail there — should be self-explanatory. Notice we no longer need to carve out testing data, the tflearn framework can do this for us.
Magnificent Seven
Our deep-learning code:
The first 5 lines define our neural ‘net’ with a sequence of tflearn functions: from tflearn.input_data to tflearn.fully_connected, to tflearn.regression. This ‘flow’ is identical to our tensorflow example: our input data has 5 features, we’ll use 32 nodes in each hidden layer and our output has 2 classes.
Next we instantiate a Deep Neural Network: tflearn.DNN with our network, with a tensorboard parameter to enable logging.
And finally we fit our model with the training data. Notice the sweet interface for the training metrics. Change the n_epochs to see impact to accuracy.
Training Step: 1999 | total loss: 0.01591 | time: 0.003s
| Adam | epoch: 1000 | loss: 0.01591 - acc: 0.9997 -- iter: 16/22
Training Step: 2000 | total loss: 0.01561 | time: 0.006s
| Adam | epoch: 1000 | loss: 0.01561 - acc: 0.9997 -- iter: 22/22
--Predictions
We now can use our model to predict output. Be sure to remove any test patterns from your training data (comment out lines containing the patterns you want to test), otherwise the model is cheating.
[[0.004509848542511463, 0.9954901337623596]]
[[0.9810173511505127, 0.018982617184519768]]Our model correctly recognizes the [1, _, _, _, 1] pattern with output [1, 0]
As a convenience when working iteratively with notebook, we reset our model’s graph by adding 2 lines directly above our model code:
By abstracting, we can focus on preparing our data and using our model to make predictions.
Tensorboard
Our tflearn framework automatically passes data to tensorboard: a visualization tool for tensorflow. Because we provided a log file with tflearn.DNN we can have a quick look.
$ tensorboard — logdir=tflearn_logs
Starting TensorBoard b’41' on port 6006
(You can navigate to http://127.0.1.1:6006)
Here we can see a graph of our ‘flow’:
And our accuracy and loss function performance:
It’s clear we don’t need as many epochs in our training to achieve solid accuracy.
Other examples
Here’s a tflearn setup for an LSTM RNN (Long-Short-Term-Memory Recurrent Neural-Net), often used to learn sequences of data with memory. Notice a different setup for the network and the tflearn.lstm, but mostly the same basic concept.
And here’s a tflearn setup for a Convolutional Neural Network, often used for image recognition. Notice again all we’re doing is providing the mathematical sequence for the network’s mathematical equations, then feeding it data.
We started by learning from code without any frameworks, this showed us precisely what was going on. No ‘black box’. Once we have a solid understanding of the underlying code, we use frameworks to simplify our work, knowing that what’s inside.
Deep-learning frameworks simplify your work by encapsulating the underlying functions necessary. As the frameworks evolve and improve, we inherit those improvements automatically, consequentially we go from ‘black box’ to ‘black boxes within a black box’.
“So let it be written, So let it be done”
