Dynamic Gesture Recognition for Sinhala Sign Language Using Pose Based Method

Abstract

"Sinhala Sign Language plays a vital role in facilitating communication for the deaf community in Sri Lanka. However, accurately recognizing and understanding the dynamic gestures involved presents significant difficulties. With the help of recent technologies, such as Deep Learning, the gap can be bridged by converting word level Sign Language gestures into text. The proposed method utilizes the Mediapipe framework to extract pose keypoints from video sequences of sign language gestures. A deep learning model based on transformers is designed and trained on a comprehensive dataset of annotated Sinhala Sign Language gestures. By capturing the unique dynamics and temporal characteristics of the gestures, the model achieves accurate recognition. Experimental evaluations demonstrate the effectiveness of the proposed method, showcasing significant improvements in dynamic gesture recognition performance compared to existing approaches. Overall accuracy of 98.75% was achieved with homegrown Sinhala Sign dataset. Furthermore, model was trained with WLASL25 dataset, and it gives 75% of accuracy on testing dataset. This is within Top-5 results. The results of this research highlight the potential impact of the pose-based method in enhancing communication and inclusivity for the deaf community in Sri Lanka. Further advancements could involve expanding the dataset, refining the model architecture, and exploring real-time applications to enhance the recognition system. "