Keras -Preprocessing Layers

In this blog I want to write a bit about the new experimental preprocessing layers in TensorFlow2.3

As we all know pre-processing is a really important step before data can be fed into a model. The reason is pretty simple, we need the inputs to be standardized so one variable being in a different scale does not completely take over the regression and messes up the model.

The old way we have been doing this is using the sklearn.preprocessing modules MinMaxScaler, StandardScaler, Normalizer etc. Btw here is an interesting blog comparing the effect of different scalars. Please do check this out.

The problem with using these scalars is that these are not saved when a model is being saved, so say if you need to implement a model (tf.keras) in JavaScript etc then you would have to re-write the whole scalar functions again in JavaScript.

Wouldn’t t it be just better if the pre-processing steps are also part of the model and we can just save the model and then load it up in JavaScript or what ever language what has support for TensorFlow.

Now that we know why we want to do, lets jump in and see how.

#Basic imports
import tensorflow as tf
import tensorflow.keras as keras
from sklearn.datasets import load_boston
import pandas as pd
import numpy as np
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings('ignore')

We are going to use the Boston Housing data from sklearn dataset for this task.

boston_data=load_boston()
data=pd.DataFrame(boston_data.data,
                 columns=boston_data.feature_names)
target=pd.DataFrame(boston_data.target)
final_boston=pd.concat([data,target],axis=1)
final_boston.rename(columns={0:'TARGET'},inplace=True)

Now we do the usual train, test splits

train, test = train_test_split(final_boston, test_size=0.2)
features = train.drop(columns='TARGET').columns
x_train = np.array(train.drop(columns='TARGET').values)
y_train = np.array(train['TARGET'].values)
x_test = np.array(test.drop(columns='TARGET').values)
y_test = np.array(test['TARGET'].values)

Here is the preprocessing ‘Normalizer’ that we use to ‘adapt’

normalizer = tf.keras.layers.experimental.preprocessing.Normalization()
normalizer.adapt(x_train)

Now we create a simple functional model using this layer

#Functional model using pre-processing layer
inputs = tf.keras.Input(shape=x_train.shape[1:])
x = normalizer(inputs)
x = tf.keras.layers.Dense(200,activation='relu') (x)
x = tf.keras.layers.Dense(100,activation='relu') (x)
x = tf.keras.layers.Dropout(0.25) (x)
x = tf.keras.layers.Dense(50,activation='relu') (x)
x = tf.keras.layers.Dense(25,activation='relu') (x)
output = tf.keras.layers.Dense(1) (x)
model = tf.keras.Model(inputs,output)
model.summary()

Model Summary

Now we do the usual compile, fit & evaluate

model.compile(optimizer='adam',loss='mean_squared_error')
model.fit(x_train,y_train,epochs=500,verbose=0)
model.evaluate(x_test,y_test)

Model Evaluate, final loss

This gives us a final loss of 17.3414, so RMSE = 4.16

The best part of all this, is that you can directly input a random x_test array and it will split out the ‘predicted’ value.

model.predict(np.array(x_test[42]).reshape(-1,13))

Predicted house price is 44.84k

y_test[42]

Actual house price 50k

We are fairly accurate, obviously this can be improved by using a better model. But to keep things simple, lets see how the same exact model without the pre-processing does.

model2 = tf.keras.Sequential([
   tf.keras.Input(shape=input_shape),
   tf.keras.layers.Dense(200,activation='relu'),
   tf.keras.layers.Dense(100,activation='relu'),
   tf.keras.layers.Dropout(0.25),
   tf.keras.layers.Dense(50,activation='relu'),
   tf.keras.layers.Dense(25,activation='relu'),
   tf.keras.layers.Dense(1)
])
model2.summary()

model2.compile(optimizer='adam',loss='mean_squared_error')
model2.fit(x_train,y_train,epochs=500,verbose=0)
model2.evaluate(x_test,y_test)

Model2 Evaluate, final loss

This gives us a final loss of 65.35, so RMSE = 8.08k

And if we try to predict the same house price with this model we get

model2.predict(np.array(x_test[42]).reshape(-1,13))

Predicted house price: 35.42k

As you can see the model with pre-processing layers had a much better predicted price and rmse.

I am sure there are much better ways of doing this but my emphasis was to incorporate the pre-processing layers into the actual model.

The github link for the full notebook is here.

Hope you enjoyed this.

Happy reading !!!

References

Module: tf.keras.layers.experimental.preprocessing

Compare the effect of different scalers on data with outliers - scikit-learn 0.23.2 documentation

Working with preprocessing layers | TensorFlow Core