Case Study: Facial Expression Recognition

Table of Contents

Introduction

The most expressive way humans display emotions is through facial expressions. Humans detect and interpret faces and facial expressions in a scene with little or no effort. An automated system could also be possible to accomplish this task by using a deep learning model that will be train based on Convolutional Neural Networks and Residual Blocks to detect facial expressions.

Problem:

  • Classify the different facial expression

Dataset:

  • The data consists of 48x48 pixel grayscale images of faces. The faces have been automatically registered so that the face is more or less centered and occupies about the same amount of space in each image. The task is to categorize each face based on the emotion shown in the facial expression in to one of seven categories (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral).

Source:

  • This dataset was prepared by Pierre-Luc Carrier and Aaron Courville, as part of an ongoing research project. They have graciously provided the workshop organizers with a preliminary version of their dataset to use for this contest.

Review

Libraries And Data Importation

In [1]:
# Import libraries
import cv2
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
import os
import PIL
from tensorflow.keras import layers
import tensorflow as tf
import pickle
from tensorflow.python.keras import *
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras import layers, optimizers
from tensorflow.keras.applications import DenseNet121
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.initializers import glorot_uniform
from tensorflow.keras.utils import plot_model
from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint, LearningRateScheduler
from IPython.display import display
from tensorflow.keras import backend as K
In [2]:
# Import data
emotion_df = pd.read_csv('emotion.csv')
In [3]:
# Check data
# Picture is represented by number based on pixel
emotion_df.head()
Out[3]:
emotion pixels
0 0 70 80 82 72 58 58 60 63 54 58 60 48 89 115 121...
1 0 151 150 147 155 148 133 111 140 170 174 182 15...
2 2 24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1...
3 2 20 17 19 21 25 38 42 42 46 54 56 62 63 66 82 1...
4 3 77 78 79 79 78 75 60 55 47 48 58 73 77 79 57 5...
In [4]:
# Check data
emotion_df['emotion'][2]
Out[4]:
2
In [5]:
# Add space # String format
emotion_df['pixels'] = emotion_df[' pixels']
In [6]:
# Check data # String format
emotion_df['pixels'][2]
Out[6]:
'24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 19 43 52 13 26 40 59 65 12 20 63 99 98 98 111 75 62 41 73 118 140 192 186 187 188 190 190 187 182 176 173 172 173 25 34 29 35 29 26 20 23 19 31 22 21 20 31 26 17 34 75 37 18 38 80 85 25 38 26 34 97 94 76 86 95 70 39 98 129 185 188 188 189 190 190 188 182 176 173 172 173 26 29 25 35 25 30 20 21 16 28 34 26 15 17 19 22 29 62 64 47 45 76 76 78 65 82 12 14 56 56 72 74 66 65 86 129 172 190 187 189 190 190 187 182 177 172 172 174 26 33 32 31 25 33 26 21 16 16 37 27 14 15 12 17 31 38 40 58 73 59 35 82 68 72 43 22 33 52 51 66 71 57 71 121 160 191 188 189 190 190 187 184 178 173 173 174 29 40 29 29 32 32 28 26 19 13 23 34 17 18 14 15 26 25 21 40 82 44 29 34 36 32 36 59 55 38 46 64 71 62 60 107 137 191 189 190 190 190 188 184 178 175 173 175 33 39 25 33 37 33 25 30 20 18 18 23 20 22 20 17 33 20 18 25 37 31 22 29 28 24 29 38 67 38 35 75 68 66 58 89 110 179 191 189 189 190 187 184 179 175 173 174 32 41 32 31 39 33 26 34 27 23 28 30 21 24 29 22 26 23 21 22 33 28 28 26 28 23 27 21 61 53 24 72 78 64 54 84 94 155 195 189 191 190 188 184 179 174 172 173 28 46 40 29 42 34 31 39 37 30 33 33 28 23 30 31 24 19 17 18 27 25 27 23 25 24 23 13 40 69 29 45 74 74 59 88 88 145 196 188 191 191 188 183 178 174 172 174 27 44 42 35 42 34 39 40 41 37 33 33 34 31 32 35 32 19 19 18 16 17 22 21 23 25 22 14 30 74 36 32 62 75 59 89 94 139 194 188 191 191 187 184 179 174 173 174 28 38 38 42 41 36 51 44 37 40 35 33 29 38 38 43 39 28 23 19 22 24 26 27 26 29 21 17 37 75 31 32 49 55 57 102 108 138 193 189 191 192 189 184 180 176 174 174 29 33 30 34 41 39 46 45 37 38 38 34 28 39 41 44 36 37 28 29 32 34 35 34 33 33 27 23 39 72 28 22 44 70 82 127 100 137 194 189 190 189 188 185 181 176 174 176 29 32 28 24 40 35 35 30 30 31 38 29 31 37 30 34 30 41 52 57 58 56 53 45 50 40 46 24 30 60 27 64 120 130 141 139 85 149 194 187 189 188 187 185 181 177 175 176 34 24 33 27 26 27 27 21 24 28 34 27 33 30 23 26 27 52 91 125 125 119 100 89 88 84 66 24 48 57 89 137 155 159 175 155 105 173 193 189 189 189 187 185 181 177 175 176 33 74 146 130 75 24 23 20 20 23 26 24 27 24 24 31 60 118 156 158 156 155 141 135 131 108 42 68 73 97 140 146 164 178 186 191 156 185 189 189 188 188 186 184 180 177 175 176 45 121 121 162 155 61 17 22 21 23 22 21 21 24 29 62 127 165 170 163 166 159 144 128 103 53 67 99 90 129 148 153 175 195 201 180 174 188 187 188 188 187 186 184 180 177 176 176 64 105 116 165 150 143 45 17 19 23 23 22 23 25 52 122 166 171 172 174 169 160 152 128 91 97 124 119 123 144 151 157 179 177 143 154 184 187 188 189 189 187 186 183 180 177 176 176 125 93 150 159 133 143 106 33 49 27 24 25 23 44 120 166 170 165 159 142 132 120 117 119 129 131 128 132 135 142 146 158 144 94 131 185 185 186 187 188 189 187 185 183 179 176 175 176 201 82 161 117 85 143 158 109 119 67 21 39 86 132 168 173 171 167 161 148 134 123 102 89 100 102 97 108 124 131 152 118 75 138 131 170 185 186 187 188 189 187 185 183 178 175 175 175 225 80 130 122 84 146 145 162 159 129 31 94 168 172 170 170 167 162 143 117 112 77 50 47 65 100 127 135 129 137 157 56 73 168 181 180 185 187 188 189 189 187 184 181 178 174 173 175 217 80 78 165 127 109 119 169 173 163 108 138 172 168 172 171 165 158 140 101 65 43 44 56 38 95 141 160 146 166 142 62 116 171 198 180 185 186 188 188 187 186 183 181 177 174 173 175 194 64 36 127 178 168 113 132 172 166 153 158 170 170 171 173 171 168 165 161 139 96 72 69 53 111 163 170 155 163 187 143 162 205 204 180 186 187 188 188 187 185 184 181 177 174 173 175 144 55 54 51 146 177 158 154 173 169 160 165 168 161 145 136 164 172 167 153 137 132 117 90 85 134 168 176 162 160 212 197 188 208 203 182 188 187 189 189 187 186 184 181 177 174 173 175 127 79 91 61 47 157 160 166 161 164 160 154 148 153 146 105 131 154 146 145 128 108 118 104 114 127 156 166 156 163 207 190 185 207 194 181 183 185 180 180 184 181 183 180 176 174 172 174 136 102 122 138 60 73 130 139 138 153 137 142 122 135 140 114 89 144 117 108 116 110 110 103 97 107 142 145 139 151 202 157 159 189 167 175 162 172 158 161 159 171 186 181 177 173 173 174 153 136 152 203 144 60 110 128 142 143 137 141 130 125 138 122 73 141 138 121 82 131 108 94 95 113 137 150 150 151 201 147 145 185 172 163 160 161 158 169 164 179 184 180 176 172 172 173 163 167 189 211 200 74 106 158 143 153 158 145 140 145 158 133 70 135 142 145 80 114 128 104 97 125 154 155 169 165 193 196 154 204 194 172 186 179 180 181 180 183 182 180 175 172 171 174 182 200 209 210 215 126 96 166 162 163 165 166 150 146 182 165 78 134 170 168 75 88 125 109 95 138 161 168 177 165 172 200 182 209 195 187 188 190 189 188 187 185 182 179 175 171 171 172 204 206 211 213 215 177 80 152 158 159 160 170 162 141 178 169 92 130 182 174 84 65 109 109 89 137 149 151 153 137 143 178 190 209 185 182 187 186 185 185 185 183 181 177 173 170 170 171 206 209 211 213 213 202 117 123 161 153 154 172 170 143 173 169 111 131 182 177 106 51 91 109 112 109 111 132 140 150 198 209 203 215 207 190 182 184 185 185 184 183 182 178 174 170 169 170 206 209 211 214 213 208 169 100 150 147 145 166 172 143 171 171 124 135 178 174 128 45 76 109 127 136 144 164 182 190 202 215 218 178 200 226 211 192 181 184 184 183 182 179 174 169 167 168 210 209 210 214 214 210 196 112 118 149 138 159 171 138 159 164 123 139 176 169 149 45 64 117 144 155 172 199 200 201 199 226 193 157 165 189 217 231 208 181 179 180 180 177 172 169 166 167 211 210 210 213 214 212 205 156 81 137 136 144 157 136 136 149 135 142 176 161 181 80 35 84 118 147 178 186 188 146 129 194 141 149 124 105 145 192 232 211 178 180 181 178 172 169 165 166 209 212 210 211 214 213 207 191 98 87 135 128 143 151 135 151 150 137 158 142 159 146 50 38 76 160 166 160 156 78 135 129 116 124 60 70 97 146 206 225 180 177 178 177 172 168 164 165 197 211 211 210 211 213 209 200 155 60 95 117 126 155 147 136 150 141 143 147 132 130 103 110 166 180 161 159 123 83 136 82 122 50 33 80 131 170 198 213 179 177 178 177 172 168 164 165 181 191 217 209 209 211 208 202 184 104 51 92 100 126 151 144 157 155 152 157 133 107 111 154 176 171 154 160 106 55 50 78 104 25 41 94 145 186 208 201 178 179 179 178 173 167 163 163 180 159 205 211 208 209 207 201 187 151 64 55 85 94 139 156 164 157 160 167 151 129 138 170 172 159 153 164 74 35 52 90 57 22 82 154 172 181 209 192 176 178 177 177 173 167 162 162 153 160 171 209 207 207 206 200 190 167 109 47 65 78 115 158 166 162 159 168 159 146 162 172 167 141 165 149 19 30 41 54 35 62 146 170 178 191 209 184 176 178 177 178 174 167 161 161 160 163 141 183 208 204 204 199 188 173 140 74 54 73 94 156 170 167 162 166 158 155 169 169 151 138 176 161 20 28 48 52 46 124 167 179 184 201 205 179 178 177 177 176 173 165 160 158 157 169 141 140 198 202 201 200 187 173 151 111 68 79 92 148 170 167 166 168 163 162 170 161 151 157 179 187 36 34 42 46 98 141 163 179 182 201 196 175 177 176 175 175 172 165 159 156 118 164 162 115 164 202 198 201 191 177 158 133 99 101 126 154 163 168 167 169 163 164 166 162 165 162 171 202 67 43 51 99 139 149 161 166 174 195 187 176 177 176 176 175 171 165 158 153 101 133 171 130 118 183 196 199 195 184 166 150 127 131 162 168 162 166 166 167 163 165 164 164 165 159 167 204 96 54 94 128 138 156 166 165 172 191 179 176 176 176 174 173 170 164 157 151 118 103 147 156 105 145 192 196 195 188 178 171 145 151 181 171 164 167 170 168 164 168 167 164 162 156 163 202 122 76 99 103 129 151 156 166 183 194 173 174 173 173 173 173 170 164 156 152 135 106 109 151 129 112 174 189 192 190 188 181 166 145 175 171 148 166 169 169 167 166 167 164 162 158 162 198 142 89 95 95 119 141 148 163 197 194 170 172 172 172 172 171 169 165 158 152 157 131 95 119 143 109 140 181 184 188 187 184 176 155 144 158 128 158 170 169 169 167 167 165 165 161 160 193 173 98 101 103 116 132 138 164 206 190 164 168 167 168 169 168 166 163 157 152 184 160 119 94 125 127 111 163 174 182 184 183 180 168 147 149 119 146 171 168 169 168 169 169 168 161 155 183 216 120 102 101 110 128 130 150 203 183 156 163 163 163 164 165 163 161 157 151 159 185 157 107 99 128 114 132 167 173 179 179 178 171 159 158 156 146 169 169 169 170 169 167 169 167 164 183 235 162 105 105 105 126 137 149 198 199 150 156 157 157 159 160 160 157 156 153 136 157 187 152 104 109 124 115 147 163 170 172 173 170 163 160 172 177 168 170 168 170 168 170 173 177 188 198 232 202 116 108 108 124 138 147 185 219 157 140 144 144 146 147 151 152 152 150 145 130 161 184 141 105 118 119 124 150 161 167 169 167 162 164 169 181 172 168 169 170 174 179 180 182 196 208 226 230 140 113 114 125 135 141 172 214 174 126 132 132 133 136 139 142 143 142'
In [7]:
# Function to convert pixel values in string format to array format
def string2array(x):
  return np.array(x.split(' ')).reshape(48, 48, 1).astype('float32')
In [8]:
# Apply the function
emotion_df['pixels'] = emotion_df['pixels'].apply(lambda x: string2array(x))
In [9]:
# Check shape
emotion_df['pixels'][2].shape
Out[9]:
(48, 48, 1)
In [10]:
# Check data
emotion_df['pixels'][2];
In [11]:
# Checking for the presence of null values in the data frame
emotion_df.isnull().sum()
Out[11]:
emotion    0
 pixels    0
pixels     0
dtype: int64
In [12]:
# Create label
label_to_text = {0:'anger', 1:'disgust', 2:'sad', 3:'happiness', 4: 'surprise'}
In [13]:
# Check image
plt.imshow(emotion_df['pixels'][0].squeeze(), cmap = 'gray')
plt.show()

Data Visualization

In [14]:
# Create List
emotions = [0,1,2,3,4]

# Create function
for i in emotions:
  data = emotion_df[emotion_df['emotion'] == i][:1]
  img = data['pixels'].item()
  img = img.reshape(48,48)
  plt.figure()
  plt.title(label_to_text[i])
  plt.imshow(img, cmap= 'gray')
In [15]:
# Create barplot
plt.figure(figsize = (10, 10))
sns.barplot(x = emotion_df.emotion.value_counts().index, y = emotion_df['emotion'].value_counts())
plt.title("Number of images per emotion")
plt.show()

Data Preprocessing

In [16]:
# Split the dataframe to features and labels
# From keras.utils import to_categorical
X = emotion_df['pixels']
X = np.stack(X, axis = 0)
X = X.reshape(24568, 48, 48, 1)

# y = to_categorical(emotion_df['emotion'])
y = pd.get_dummies(emotion_df['emotion'])

# Show data dimensions
print(X.shape, y.shape)

(24568, 48, 48, 1) (24568, 5)
In [17]:
# Transform data into arrays
X = np.array(X)
y = np.array(y)
In [18]:
# Import lirabries
from sklearn.model_selection import train_test_split

# spliting the dataframe in to train,test and validation data frames
X_train, X_Test, y_train, y_Test = train_test_split(X,y,test_size = 0.1, shuffle = True)
X_val, X_Test, y_val, y_Test = train_test_split(X_Test,y_Test, test_size = 0.5, shuffle = True)
In [19]:
# Check train dataset
print(X_train.shape, y_train.shape)

(22111, 48, 48, 1) (22111, 5)
In [20]:
# Check validation dataset
print(X_val.shape, y_val.shape)

(1228, 48, 48, 1) (1228, 5)
In [21]:
# Check test dataset
print(X_Test.shape, y_Test.shape)

(1229, 48, 48, 1) (1229, 5)
In [22]:
# Normalize images
X_train = X_train/255
X_val   = X_val /255
X_Test  = X_Test/255
In [23]:
# Data augmentation
train_datagen = ImageDataGenerator(
rotation_range = 15,
    width_shift_range = 0.1,
    height_shift_range = 0.1,
    shear_range = 0.1,
    zoom_range = 0.1,
    fill_mode = "nearest"
)

CNN And Residual Neural Network

In [24]:
# Create funchtion
def res_block(X, filter, stage):

  # Convolutional_block
  X_copy = X

  f1 , f2, f3 = filter

  # Main Path
  X = Conv2D(f1, (1,1),strides = (1,1), name ='res_'+str(stage)+'_conv_a', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = MaxPool2D((2,2))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_a')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f2, kernel_size = (3,3), strides =(1,1), padding = 'same', name ='res_'+str(stage)+'_conv_b', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_b')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_conv_c', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_c')(X)


  # Short path
  X_copy = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_conv_copy', kernel_initializer= glorot_uniform(seed = 0))(X_copy)
  X_copy = MaxPool2D((2,2))(X_copy)
  X_copy = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_copy')(X_copy)

  # ADD
  X = Add()([X,X_copy])
  X = Activation('relu')(X)

  # Identity Block 1
  X_copy = X


  # Main Path
  X = Conv2D(f1, (1,1),strides = (1,1), name ='res_'+str(stage)+'_identity_1_a', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_1_a')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f2, kernel_size = (3,3), strides =(1,1), padding = 'same', name ='res_'+str(stage)+'_identity_1_b', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_1_b')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_identity_1_c', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_1_c')(X)

  # ADD
  X = Add()([X,X_copy])
  X = Activation('relu')(X)

  # Identity Block 2
  X_copy = X


  # Main Path
  X = Conv2D(f1, (1,1),strides = (1,1), name ='res_'+str(stage)+'_identity_2_a', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_2_a')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f2, kernel_size = (3,3), strides =(1,1), padding = 'same', name ='res_'+str(stage)+'_identity_2_b', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_2_b')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_identity_2_c', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_2_c')(X)

  # ADD
  X = Add()([X,X_copy])
  X = Activation('relu')(X)

  return X
In [25]:
# Input tensor shape
input_shape = (48, 48, 1)

# Input tensor shape
X_input = Input(input_shape)

# Zero-padding
X = ZeroPadding2D((3, 3))(X_input)

# 1 - stage
X = Conv2D(64, (7, 7), strides= (2, 2), name = 'conv1', kernel_initializer= glorot_uniform(seed = 0))(X)
X = BatchNormalization(axis =3, name = 'bn_conv1')(X)
X = Activation('relu')(X)
X = MaxPooling2D((3, 3), strides= (2, 2))(X)

# 2 - stage
X = res_block(X, filter= [64, 64, 256], stage= 2)

# 3 - stage
X = res_block(X, filter= [128, 128, 512], stage= 3)

# 4 - stage
# X = res_block(X, filter= [256, 256, 1024], stage= 4)

# Average Pooling
X = AveragePooling2D((2, 2), name = 'Averagea_Pooling')(X)

# Final layer
X = Flatten()(X)
X = Dense(5, activation = 'softmax', name = 'Dense_final', kernel_initializer= glorot_uniform(seed=0))(X)

# Model architecture
model_emotion = Model( inputs= X_input, outputs = X, name = 'Resnet18')

# Check model summary
model_emotion.summary()

Model: "Resnet18"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 48, 48, 1)]  0                                            
__________________________________________________________________________________________________
zero_padding2d (ZeroPadding2D)  (None, 54, 54, 1)    0           input_1[0][0]                    
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 24, 24, 64)   3200        zero_padding2d[0][0]             
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 24, 24, 64)   256         conv1[0][0]                      
__________________________________________________________________________________________________
activation (Activation)         (None, 24, 24, 64)   0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
max_pooling2d (MaxPooling2D)    (None, 11, 11, 64)   0           activation[0][0]                 
__________________________________________________________________________________________________
res_2_conv_a (Conv2D)           (None, 11, 11, 64)   4160        max_pooling2d[0][0]              
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, 5, 5, 64)     0           res_2_conv_a[0][0]               
__________________________________________________________________________________________________
bn_2_conv_a (BatchNormalization (None, 5, 5, 64)     256         max_pooling2d_1[0][0]            
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 5, 5, 64)     0           bn_2_conv_a[0][0]                
__________________________________________________________________________________________________
res_2_conv_b (Conv2D)           (None, 5, 5, 64)     36928       activation_1[0][0]               
__________________________________________________________________________________________________
bn_2_conv_b (BatchNormalization (None, 5, 5, 64)     256         res_2_conv_b[0][0]               
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 5, 5, 64)     0           bn_2_conv_b[0][0]                
__________________________________________________________________________________________________
res_2_conv_copy (Conv2D)        (None, 11, 11, 256)  16640       max_pooling2d[0][0]              
__________________________________________________________________________________________________
res_2_conv_c (Conv2D)           (None, 5, 5, 256)    16640       activation_2[0][0]               
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, 5, 5, 256)    0           res_2_conv_copy[0][0]            
__________________________________________________________________________________________________
bn_2_conv_c (BatchNormalization (None, 5, 5, 256)    1024        res_2_conv_c[0][0]               
__________________________________________________________________________________________________
bn_2_conv_copy (BatchNormalizat (None, 5, 5, 256)    1024        max_pooling2d_2[0][0]            
__________________________________________________________________________________________________
add (Add)                       (None, 5, 5, 256)    0           bn_2_conv_c[0][0]                
                                                                 bn_2_conv_copy[0][0]             
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 5, 5, 256)    0           add[0][0]                        
__________________________________________________________________________________________________
res_2_identity_1_a (Conv2D)     (None, 5, 5, 64)     16448       activation_3[0][0]               
__________________________________________________________________________________________________
bn_2_identity_1_a (BatchNormali (None, 5, 5, 64)     256         res_2_identity_1_a[0][0]         
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 5, 5, 64)     0           bn_2_identity_1_a[0][0]          
__________________________________________________________________________________________________
res_2_identity_1_b (Conv2D)     (None, 5, 5, 64)     36928       activation_4[0][0]               
__________________________________________________________________________________________________
bn_2_identity_1_b (BatchNormali (None, 5, 5, 64)     256         res_2_identity_1_b[0][0]         
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 5, 5, 64)     0           bn_2_identity_1_b[0][0]          
__________________________________________________________________________________________________
res_2_identity_1_c (Conv2D)     (None, 5, 5, 256)    16640       activation_5[0][0]               
__________________________________________________________________________________________________
bn_2_identity_1_c (BatchNormali (None, 5, 5, 256)    1024        res_2_identity_1_c[0][0]         
__________________________________________________________________________________________________
add_1 (Add)                     (None, 5, 5, 256)    0           bn_2_identity_1_c[0][0]          
                                                                 activation_3[0][0]               
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 5, 5, 256)    0           add_1[0][0]                      
__________________________________________________________________________________________________
res_2_identity_2_a (Conv2D)     (None, 5, 5, 64)     16448       activation_6[0][0]               
__________________________________________________________________________________________________
bn_2_identity_2_a (BatchNormali (None, 5, 5, 64)     256         res_2_identity_2_a[0][0]         
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 5, 5, 64)     0           bn_2_identity_2_a[0][0]          
__________________________________________________________________________________________________
res_2_identity_2_b (Conv2D)     (None, 5, 5, 64)     36928       activation_7[0][0]               
__________________________________________________________________________________________________
bn_2_identity_2_b (BatchNormali (None, 5, 5, 64)     256         res_2_identity_2_b[0][0]         
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 5, 5, 64)     0           bn_2_identity_2_b[0][0]          
__________________________________________________________________________________________________
res_2_identity_2_c (Conv2D)     (None, 5, 5, 256)    16640       activation_8[0][0]               
__________________________________________________________________________________________________
bn_2_identity_2_c (BatchNormali (None, 5, 5, 256)    1024        res_2_identity_2_c[0][0]         
__________________________________________________________________________________________________
add_2 (Add)                     (None, 5, 5, 256)    0           bn_2_identity_2_c[0][0]          
                                                                 activation_6[0][0]               
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 5, 5, 256)    0           add_2[0][0]                      
__________________________________________________________________________________________________
res_3_conv_a (Conv2D)           (None, 5, 5, 128)    32896       activation_9[0][0]               
__________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)  (None, 2, 2, 128)    0           res_3_conv_a[0][0]               
__________________________________________________________________________________________________
bn_3_conv_a (BatchNormalization (None, 2, 2, 128)    512         max_pooling2d_3[0][0]            
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 2, 2, 128)    0           bn_3_conv_a[0][0]                
__________________________________________________________________________________________________
res_3_conv_b (Conv2D)           (None, 2, 2, 128)    147584      activation_10[0][0]              
__________________________________________________________________________________________________
bn_3_conv_b (BatchNormalization (None, 2, 2, 128)    512         res_3_conv_b[0][0]               
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 2, 2, 128)    0           bn_3_conv_b[0][0]                
__________________________________________________________________________________________________
res_3_conv_copy (Conv2D)        (None, 5, 5, 512)    131584      activation_9[0][0]               
__________________________________________________________________________________________________
res_3_conv_c (Conv2D)           (None, 2, 2, 512)    66048       activation_11[0][0]              
__________________________________________________________________________________________________
max_pooling2d_4 (MaxPooling2D)  (None, 2, 2, 512)    0           res_3_conv_copy[0][0]            
__________________________________________________________________________________________________
bn_3_conv_c (BatchNormalization (None, 2, 2, 512)    2048        res_3_conv_c[0][0]               
__________________________________________________________________________________________________
bn_3_conv_copy (BatchNormalizat (None, 2, 2, 512)    2048        max_pooling2d_4[0][0]            
__________________________________________________________________________________________________
add_3 (Add)                     (None, 2, 2, 512)    0           bn_3_conv_c[0][0]                
                                                                 bn_3_conv_copy[0][0]             
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 2, 2, 512)    0           add_3[0][0]                      
__________________________________________________________________________________________________
res_3_identity_1_a (Conv2D)     (None, 2, 2, 128)    65664       activation_12[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_a (BatchNormali (None, 2, 2, 128)    512         res_3_identity_1_a[0][0]         
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 2, 2, 128)    0           bn_3_identity_1_a[0][0]          
__________________________________________________________________________________________________
res_3_identity_1_b (Conv2D)     (None, 2, 2, 128)    147584      activation_13[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_b (BatchNormali (None, 2, 2, 128)    512         res_3_identity_1_b[0][0]         
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 2, 2, 128)    0           bn_3_identity_1_b[0][0]          
__________________________________________________________________________________________________
res_3_identity_1_c (Conv2D)     (None, 2, 2, 512)    66048       activation_14[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_c (BatchNormali (None, 2, 2, 512)    2048        res_3_identity_1_c[0][0]         
__________________________________________________________________________________________________
add_4 (Add)                     (None, 2, 2, 512)    0           bn_3_identity_1_c[0][0]          
                                                                 activation_12[0][0]              
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 2, 2, 512)    0           add_4[0][0]                      
__________________________________________________________________________________________________
res_3_identity_2_a (Conv2D)     (None, 2, 2, 128)    65664       activation_15[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_a (BatchNormali (None, 2, 2, 128)    512         res_3_identity_2_a[0][0]         
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 2, 2, 128)    0           bn_3_identity_2_a[0][0]          
__________________________________________________________________________________________________
res_3_identity_2_b (Conv2D)     (None, 2, 2, 128)    147584      activation_16[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_b (BatchNormali (None, 2, 2, 128)    512         res_3_identity_2_b[0][0]         
__________________________________________________________________________________________________
activation_17 (Activation)      (None, 2, 2, 128)    0           bn_3_identity_2_b[0][0]          
__________________________________________________________________________________________________
res_3_identity_2_c (Conv2D)     (None, 2, 2, 512)    66048       activation_17[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_c (BatchNormali (None, 2, 2, 512)    2048        res_3_identity_2_c[0][0]         
__________________________________________________________________________________________________
add_5 (Add)                     (None, 2, 2, 512)    0           bn_3_identity_2_c[0][0]          
                                                                 activation_15[0][0]              
__________________________________________________________________________________________________
activation_18 (Activation)      (None, 2, 2, 512)    0           add_5[0][0]                      
__________________________________________________________________________________________________
Averagea_Pooling (AveragePoolin (None, 1, 1, 512)    0           activation_18[0][0]              
__________________________________________________________________________________________________
flatten (Flatten)               (None, 512)          0           Averagea_Pooling[0][0]           
__________________________________________________________________________________________________
Dense_final (Dense)             (None, 5)            2565        flatten[0][0]                    
==================================================================================================
Total params: 1,174,021
Trainable params: 1,165,445
Non-trainable params: 8,576
__________________________________________________________________________________________________
In [26]:
# Compile the network
model_emotion.compile(optimizer = "Adam", loss = "categorical_crossentropy", metrics = ["accuracy"])
In [27]:
# Using early stopping to exit training if validation loss is not decreasing even after certain epochs (patience)
earlystopping = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 1, patience = 20)

# Save the best model with lower validation loss
checkpointer = ModelCheckpoint(filepath = "FacialExpression_weights.hdf5", verbose = 1, save_best_only=True)
In [28]:
# Train the model
history = model_emotion.fit(train_datagen.flow(X_train, y_train, batch_size = 64),
    validation_data = (X_val, y_val), steps_per_epoch = len(X_train) // 64,
    epochs= 10, callbacks = [checkpointer, earlystopping])

Epoch 1/10
345/345 [==============================] - ETA: 0s - loss: 1.4074 - accuracy: 0.4287
Epoch 00001: val_loss improved from inf to 1.83491, saving model to FacialExpression_weights.hdf5
345/345 [==============================] - 406s 1s/step - loss: 1.4074 - accuracy: 0.4287 - val_loss: 1.8349 - val_accuracy: 0.3746
Epoch 2/10
345/345 [==============================] - ETA: 0s - loss: 1.1733 - accuracy: 0.5310
Epoch 00002: val_loss improved from 1.83491 to 1.35695, saving model to FacialExpression_weights.hdf5
345/345 [==============================] - 433s 1s/step - loss: 1.1733 - accuracy: 0.5310 - val_loss: 1.3569 - val_accuracy: 0.4397
Epoch 3/10
345/345 [==============================] - ETA: 0s - loss: 1.0428 - accuracy: 0.5833
Epoch 00003: val_loss improved from 1.35695 to 1.30362, saving model to FacialExpression_weights.hdf5
345/345 [==============================] - 434s 1s/step - loss: 1.0428 - accuracy: 0.5833 - val_loss: 1.3036 - val_accuracy: 0.5049
Epoch 4/10
345/345 [==============================] - ETA: 0s - loss: 0.9430 - accuracy: 0.6218
Epoch 00004: val_loss improved from 1.30362 to 0.95248, saving model to FacialExpression_weights.hdf5
345/345 [==============================] - 545s 2s/step - loss: 0.9430 - accuracy: 0.6218 - val_loss: 0.9525 - val_accuracy: 0.6197
Epoch 5/10
345/345 [==============================] - ETA: 0s - loss: 0.8948 - accuracy: 0.6464
Epoch 00005: val_loss did not improve from 0.95248
345/345 [==============================] - 379s 1s/step - loss: 0.8948 - accuracy: 0.6464 - val_loss: 1.1198 - val_accuracy: 0.5383
Epoch 6/10
345/345 [==============================] - ETA: 0s - loss: 0.8491 - accuracy: 0.6635
Epoch 00006: val_loss did not improve from 0.95248
345/345 [==============================] - 331s 958ms/step - loss: 0.8491 - accuracy: 0.6635 - val_loss: 1.2222 - val_accuracy: 0.5440
Epoch 7/10
345/345 [==============================] - ETA: 0s - loss: 0.8260 - accuracy: 0.6723
Epoch 00007: val_loss did not improve from 0.95248
345/345 [==============================] - 324s 938ms/step - loss: 0.8260 - accuracy: 0.6723 - val_loss: 1.0931 - val_accuracy: 0.5831
Epoch 8/10
345/345 [==============================] - ETA: 0s - loss: 0.7942 - accuracy: 0.6892
Epoch 00008: val_loss improved from 0.95248 to 0.94036, saving model to FacialExpression_weights.hdf5
345/345 [==============================] - 315s 913ms/step - loss: 0.7942 - accuracy: 0.6892 - val_loss: 0.9404 - val_accuracy: 0.6458
Epoch 9/10
345/345 [==============================] - ETA: 0s - loss: 0.7675 - accuracy: 0.6976
Epoch 00009: val_loss improved from 0.94036 to 0.93946, saving model to FacialExpression_weights.hdf5
345/345 [==============================] - 318s 921ms/step - loss: 0.7675 - accuracy: 0.6976 - val_loss: 0.9395 - val_accuracy: 0.6360
Epoch 10/10
345/345 [==============================] - ETA: 0s - loss: 0.7566 - accuracy: 0.7082
Epoch 00010: val_loss did not improve from 0.93946
345/345 [==============================] - 410s 1s/step - loss: 0.7566 - accuracy: 0.7082 - val_loss: 1.0667 - val_accuracy: 0.5546
In [29]:
# Saving the model architecture to json file for future use
model_json = model_emotion.to_json()
with open("Emotion-model.json","w") as json_file:
  json_file.write(model_json)

Model Evaluation

In [30]:
# Import model architecture
with open('Emotion-model.json', 'r') as json_file:
    json_savedModel= json_file.read()
    
# load the model architecture 
model_emotion = tf.keras.models.model_from_json(json_savedModel)
model_emotion.load_weights('FacialExpression_weights.hdf5')
model_emotion.compile(optimizer = "Adam", loss = "categorical_crossentropy", metrics = ["accuracy"])
In [31]:
# Check test accuracy
score = model_emotion.evaluate(X_Test, y_Test)
print('Test Accuracy: {}'.format(score[1]))

39/39 [==============================] - 3s 82ms/step - loss: 0.9336 - accuracy: 0.6485
Test Accuracy: 0.6484947204589844
In [32]:
# Check dictionary
history.history.keys()
Out[32]:
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
In [33]:
# Check evaluation metrics
accuracy = history.history['accuracy']
val_accuracy = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
In [34]:
# Number of iterations
epochs = range(len(accuracy))

# Create plot for accuracy
plt.plot(epochs, accuracy, 'bo', label='Training Accuracy')
plt.plot(epochs, val_accuracy, 'b', label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.show()
In [35]:
# Create plot for validation loss
plt.plot(epochs, loss, 'ro', label='Training loss')
plt.plot(epochs, val_loss, 'r', label='Validation loss')
plt.title('Training and Validation loss')
plt.legend()
plt.show()
In [36]:
# Predicted_classes = model.predict_classes(X_test)
predicted_classes = np.argmax(model_emotion.predict(X_Test), axis=-1)
y_true = np.argmax(y_Test, axis=-1)
In [37]:
# Check data dimension
y_true.shape
Out[37]:
(1229,)
In [38]:
# import library
from sklearn.metrics import confusion_matrix

# Confusion matrix
cm = confusion_matrix(y_true, predicted_classes)
plt.figure(figsize = (10, 10))
sns.heatmap(cm, annot = True, cbar = False)
plt.show()
In [39]:
# Length
L = 5

# Width
W = 5

fig, axes = plt.subplots(L, W, figsize = (24, 24))
axes = axes.ravel()

for i in np.arange(0, L*W):
    axes[i].imshow(X_Test[i].reshape(48,48), cmap = 'gray')
    axes[i].set_title('Prediction = {}\n True = {}'.format(label_to_text[predicted_classes[i]], label_to_text[y_true[i]]))
    axes[i].axis('off')

plt.subplots_adjust(wspace = 1)
plt.show()
In [40]:
# Import lirabry
from sklearn.metrics import classification_report

# Check classification report
print(classification_report(y_true, predicted_classes))

              precision    recall  f1-score   support

           0       0.73      0.21      0.33       243
           1       0.00      0.00      0.00        33
           2       0.62      0.61      0.62       312
           3       0.63      0.92      0.75       456
           4       0.77      0.72      0.74       185

    accuracy                           0.65      1229
   macro avg       0.55      0.49      0.49      1229
weighted avg       0.65      0.65      0.61      1229

Conclusion

It can be observed that the photos showing disgust has the lowest accuracy score it is because the data was imbalance. The solution to that is to add more examples by generating more augmented images or adjusting the weights of disgust photos. the model was able to obtained 65% accuracy with only 10 iterations. There are several related problems: detection of an image segment as a face, extraction of the facial expression information, and classification of the expression.This could be further improve by tuning the model hyper parameters and increasing the training iterations.