Overview / Usage

While deaf people from different language communities can communicate with each other without difficulty in SASL (South African Sign Language), they cannot understand "sign language" interpreters unless they have been schooled in the manually coded language used by the interpreter. With the growing adoption of assistive technologies, deaf and visually impaired people need to be able to communicate naturally with their network, regardless of whether the second person has expertise on sign language, especially under cases of video consultations with their health practitioner, Educators or be it friends & family . We propose a deep neural network for the prediction of South African Sign Language(SASL) and gesture recognition to direct standard English text translation in natural video sequences using CPU. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos.

Methodology / Approach

Data Centric AI

_It is crucial to have as much, high quality and accurate data as possible for improved model accuracy and performance, therefore training data consists two types of hand color and depth images. __The OpenGesture Dataset was collected using Intel Realsense D435 by the ABI Data Engineering Team Africa._The dataset has ten digits (0-9) sign language classes. Each gesture is repeated 30 times by two independent sample agents.

Model Centric AI

We apply transfer learning technique of which is a model that has already been trained on a related task and reusing it in a new model. OpenGesture demonstrates how to build a Keras model for classifying ten signs of gestures by using a pre -trained TF2 SavedModel from TensorFlow Hub for image feature extraction, trained on the much larger and more general ImageNet dataset. There are multiple possible models to try (more are provided on the OpenGesture Colab).

Technologies Used

TensorFlow Hub

Repository

https://github.com/AfricaMachineIntelligence/OpenGesture

Collaborators