VMouse: A Real-time Gesture-Based Virtual Mouse using Hand Landmark Model with OpenCV and Python

Abstract

Sign language is the primary way for those deaf individuals to convey themselves. In real life, many people, even some deaf individuals, do not understand sign language. This research aims to develop a sign language translator using computer vision technology which helps to minimize the communication gap between normal s and deaf individuals. SignBridge mainly focuses on the user's hand gestures and body pose. SignBridge uses the MediaPipe Holistic to extract the key points of the user’s gestures and pose and perform real-time detection using OpenCV. It is trained using the Sequential model, which consists of the Long Short Term Memory (LSTM) model and the Dense layer with the set of videos as the training data stored in NumPy. After training, SignBridge can perform the prediction of the signs and show the result to the user by displaying the text on the top of the interface. The SignBridge system will be implemented on the laptop with a webcam to capture the gestures and pose of the user. A series of data analyses and comparisons have been conducted to determine the optimal model for the prediction based on four categories: alphabets, numbers, basic gestures, and a combination of three categories. With the comparison result, SignBridge successfully gained the highest accuracy among the models, and the overall accuracy will be in the range of 96.97 percent to 100 percent.