Autoencoders in TensorFlow — part 1
In this blog I want to write a bit about Autoencoders and how they can be helpful for dimensionality reduction, feature extraction. In this blog we will look at what Autoencoders and how they can be helpful with feature extraction, we take the fashion mnist example to do this.
In the next blog, we look at how to create various Autoencoders for Convolutions, Recurrent & other types. So without further ado lets jump in.
Autoencoders: An Autoencoder is a neural network that seeks to learn dense representations of the input data by reducing the dimensionality.
Now what exactly do I mean by that, imagine a fashion mnist sample, as you might know this is a 28x28 pixel grayscale image. Now imagine for some reason we only got half the pixels (think hard disk failure, internet failure or just some random virus corrupting your image). Would we be able to construct back the original image?
As you might have guessed the answer is yes, using Autoencoders. Before we get into the code, I want to take a second a give a huge shout out to Aurelien Geron for writing the Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow book. Most of the code is inspired from this book, I made a few tweaks here and there but I cannot emphasize enough how good this book this.
Here is the code on how to do this.
#General imports
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline#Load Datasetmnist = tf.keras.datasets.fashion_mnist
(training_images, training_labels), (test_images, test_labels)
= mnist.load_data()#Scale
X_train=training_images / 255.0
X_valid=test_images/255.0
First we define the encoder stack
encoder_stack = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=[28,28]),
tf.keras.layers.Dense(100,activation='relu'),
tf.keras.layers.Dense(30,activation='relu')
])This takes the 28x28 image, flattens it and then applies deep learning and tries to extract features that are fit on only 30 neurons.
Now we do the exact opposite on the decoder stack
decoder_stack = tf.keras.models.Sequential([
tf.keras.layers.Dense(100,activation='selu',input_shape=[30]),
tf.keras.layers.Dense(28*28,activation='sigmoid'),
tf.keras.layers.Reshape([28,28])
])Here we are literally doing the exact opposite of what we did before. We read the 30 neurons input in the first layer and expand that to 100, next we expand that further to 784 (28x28) and finally reshape the output to 28x28 pixels.
Here is the final “AutoEncoder” model that combines both of them.
AE = tf.keras.models.Sequential([encoder_stack,decoder_stack])AE.compile(loss='binary_crossentropy',optimizer='adam')
Now we train this model, do note this training would need both X_train as input and output.
history = AE.fit(X_train,X_train,epochs=10,
verbose=2,validation_data=(X_valid,X_valid))
Finally we get the following results as our autoencoder results.
Now, lets see how our autoencoder learns if we drop 50% of the pixels.
encoder_stack_Drop50 = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=[28,28]),
tf.keras.layers.Dropout(0.5), #50% drop of pixels
tf.keras.layers.Dense(100,activation='relu'),
tf.keras.layers.Dense(30,activation='relu')
])Now using this encoder stack, let create a new autoencoder and train it.
AE2 = tf.keras.models.Sequential(
[encoder_stack_Drop50,decoder_stack])AE2.compile(loss='binary_crossentropy',optimizer='adam')
And here is the training
history2 = AE2.fit(X_train,X_train,epochs=10,
verbose=2,validation_data=(X_valid,X_valid))
Final results using this model.
As you can see the results are pretty good esp. when you consider how we are dropping 50% of the data in the first layer. If we think further we see how this could help us when dealing with large datasets and thousands of features.
We will look into different types of AutoEncoders in the next blog.
Happy reading !!!
References:
Book: Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow by Aurelien Geron
