Overview / Usage

OpenGesture skill seeks to simplify the process of learning and understand the sign language and gesture recognition, which is normally a communication barrier to some.

The OpenGesture Skill uses a model built upon an Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modelling hand motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified hand motion features, which simplifies the task of prediction.

Alexa handles the Speech Recognition using a custom built skill Speech-To-Sign Language translation which recognizing the words being spoken, regardless of whom the speaker is. The OpenGesture skill for Alexa will perform the recognition process through matching the parameter set of the input speech with the stored templates to finally display the sign language in video format.

Methodology / Approach

OpenGesture Skill is an inference-based application supercharged with power-efficient Intel® processors and Intel® Processor Graphics on a laptop. Based on OpenVINO inference engine backend is used by default since OpenCV 3.4.2 (OpenVINO 2018.R2) when OpenCV is built with the Inference engine support, so the call above is not necessary. Also, the Inference engine backend is the only available option (also enabled by default) when the loaded model is represented in OpenVINO™ Model Optimizer format. OpenGesture Skill is built .NET Framework and AWS Lambda to effectively handle NLP workloads.

Technologies Used

ZenBook UX430
Intel Core i7 8th Gen
16 GB DDR4 RAM
512 SSD Storage
Intel RealSense D435
Ubuntu 16.04 LTS
IntelOpenVINO Toolkit
Realsense SDK 2
OpenCV
AWS Lambda
Visual Studio
.NET Framework
Alexa Skills Kit

Repository

https://github.com/TebogoNakampe/OpenGesture

Collaborators