Overview / Usage

Aim:
To develop and implement algorithms for identifying the emotions expressed by a person based on his / her face images in a real time video. The problem lies in detecting and tracking of face images of a person in a video to identify the best instance to identify the facial expression of the person. Further, it also lies in developing a machine learning algorithm to label the instance of a face image by one of the predetermined emotional expressions such as happy, angry, neutral, sad, and surprise.

Methodology / Approach

Dataset: A video of Friends TV series that i downloaded from youtube was labelled by me using Intel's CVAT tool for video annotation. Extracted all the labeled images from given two videos along with their labels. Other than that I used Kaggle’s Face Emotion Recognition data 2013. The kaggle data is actually in csv format, so I wrote a script to convert pixels to image so as to streamline my overall data. I then generated a csv with a path to all the images and their labels. After that I split the dataset 70% training, of the remaining 30%, 33% was used for validation and 66% was used as test data.

Model: I chose VGGnet as my model for training. I used VGGnet because it gave a good trade-off between accuracy and training time. It is a comparatively smaller CNN and gives considerably good accuracy.

Training: Total number of images= 41350. This was a very good amount of data, but as it is said the more the merrier, and so I augmented the data. Augmentation would also allow my model to better generalize the features of each emotion. I used ImageDataGenerator from keras to augment the data. I stopped my training at the 659th epoch as my model had saturated and wasn’t able to learn more. It gave 73% training accuracy and 70% test accuracy.

Real Time emotion recognition: My first basic approach for detecting faces and classifying emotion was to use haarcascade. It worked well using my laptop's webcam gave almost accurate real-time results. But when I ran the same on three videos of Test dataset provided, I found out that majority of faces weren’t getting detected. Also frontalfacehaarcascade.xml wouldn’t even recognise the face even if there is slight tilt in the face. Here’s when I read about other real-time face detection techniques and came across MTCNN(Multi-task Cascaded Convolutional Networks). I integrated MTCNN to work with my code. It definitely was a great improvement over haarcascade method but it still fails to detect some faces efficiently like those faces with a lot of beared. I wish I had time to fine tune it but as of now, I used pre-trained MTCNN for detection of faces.

Technologies Used

I used DevCloud for training the network.
I used Intel's CVAT annotation tool which is one of the easiest to use tool for annotating Images and especially Videos.

Other links

• Face Detection
• Face recogniotion