Motus- Emotion Recognition Music Player
A one of a kind program that utilizes Intel OpenVINO Toolkit and Spotify to deliver song recommendations based on facial mood cues. ...learn more
Project status: Concept
Overview / Usage
Spotify has over 30 million songs available and nearly 4 billion playlists to stream. With the selection of millions of songs and playlists listeners often struggle to determine what songs or playlist they want to listen to. People are inundated with nearly 35,000 choices a day such as what to wear or eat for dinner. Even if the search is narrowed to a playlist generated by mood there are still at least thousands from chill, to comedy, to focus. For music listeners selecting a simple playlist to listen to while you complete homework can easily become time consuming and tiresome. Currently, no applications exist to automatically select a playlist on the basis of facial expressions.
Our program Motus removes the hassle of selecting a playlist based on mood. Noah, a high school senior, often struggles with choosing a playlist to listen to while he prepares for the upcoming school day. Instead of browsing aimlessly through a vast amount of playlists curated to specific moods. Using Motus Noah can take a quick selfie expressing his mood and our program will quickly suggest a playlist based on his facial expression that reflects his mood. Trading the normally gruesome playlist selecting process with a quick selfie matching his mood with a carefully selected playlist to set Noah up for an exceptional day.
Motus is a one of a kind program, as it uses the Intel OpenVino Toolkit and Spotify to deliver song recommendations based on facial moods. The application uses a pre-trained model in OpenVINO that can classify 5 different emotions: happy, sad, angry, surprised, neutral. Motus also utilizes two pre-trained models _emotions-recognition-0003 _and _face-detection-adas-0001 _available in the OpenVINO Open Model Zoo.
Motus leverages the Face Detection Demo that's available in the OpenVINO 'demos' library. The demo will have to be arranged and modified. The location of the Demo's source code is here: /opt/intel/openvino_2020.4.287/inference_engine/demos/interactive_face_detection_demo.
Motus could become an indispensable program if integrated with Spotify or Facebook. Some of Spotify’s growth is attributed to its integration within other apps like Google Maps, Tinder, Messenger, and Discord. According to Goodwater Capital, Spotify saw the largest growth after integration with Uber allowing for ride-sharing users to control the music during their car ride within the app. Facebook Portal launched two years ago but has had slow success in the smart display market considering its competitors Google Hub and Amazon Echo. David Watkins, a market analyst forecasted that Facebook portal will account for just 4% of smart-display shipments in 2019. If integrated Motus would indefinitely be a staple feature for Facebook Portal or Spotify increasing sales or monthly active listeners for Spotify.
Methodology / Approach
The purpose of this project is to display the functionality of the OpenVINO Toolkit and its use of the Model Optimizer and Inference Engine. The Vision Inference in OpenVINO offers faster inference than other frameworks, and can be manipulated for different real world applications.
When approaching this idea, we wanted to ensure that it was feasible with the current pre-trained models in the OpenVINO Toolkit, while also being practical and useful in the real world. It was determined that the `face-detection-adas-0001` and `emotions-recognition-retail-0003` models would be best suited for this project. Then, a strategy was developed in the creation of our project, leading to the flowchart attached in the Image Gallery.
Once a strategy was developed, we began to enhance our experience with the OpenVINO Toolkit, understanding how the Model Optimizer works and how to run demos. Then, we began modifying the source code of the `interactive_face_detection_demo` program (to help extract relevant data for song recommendation). A run file was created to help make the application easier to run, and a python script was written to parse through the data.
To accompany our work with the OpenVINO Toolkit, we also employed the Spotify API to get music data. We created a 500+ song playlist to represent the Motus song library and got the features of those songs that correspond to mood using requests to the Spotify API. Next we ran all of the songs and their mood features through a classification model that we trained using Tensorflow to map each song to an emotion.
The emotion data from our OpenVINO model and the emotion data from our song classification model were integrated to create our product: Motus.
Teamwork was a huge contributor to the completion of this project. A lot of brainstorming was done, and we faced many challenges and bugs, however with each other’s collaborations we were able to come up with a working prototype.
Repository
https://github.com/omarhassan91/motus
Other links
Collaborators
There are no people to show.