Overview / Usage

In conjunction with the AI assistant we are creating, which does speech to text transcription, the hand gesture recognition model will be used to execute nonverbal commands that can be recognized and detected through the RealSense camera.

Methodology / Approach

We decided which gestures we needed/wanted to execute commands. For example, the "hush" gesture - the index finger to the lips, was chosen to mute the audio.

Then, because the gestures we wanted were not in one RGBD dataset, we had to combine gestures selected from different datasets with different file and formatting types together. The end result should be a dataset with uniform resolution and file types.

Afterwards, we will run a multi-stage CNN used to regress the 3D joint locations. We are also currently looking at work done with manifolds, since the point cloud or depth data for  the region of 3D space occupied by a hand should be a half manifold, in the sense that no matter what gesture the hand is making, it is a real world 3D shape and thus must be manifold. So, rotating and scaling the manifold, combined with moving finger joints, are the only transformations of hands that can be applied in the real world. This makes manifolds potentially attractive from a gesture recognition perspective.

Technologies Used

Python
OpenCV
Tensorflow
Keras

Intel AI DevCloud
Intel NUC
Intel RealSense Depth Camera D415
Intel OpenVino