Enhance Learning American Sign Language by Gamification: Bridging Computer Vision and Games

Abstract

"Sign language is the means of communication for deaf people due to their hearing impairments. Deaf people often find it difficult to adapt in society and carry out daily activities such as shopping due to communication issues. Sign language is not a popular mode of communication in the society and therefore, the deaf people will have to spend much energy or use sign language translators while conversing with non-sign language speakers.

In this research, a responsive web based application is proposed with gamification approaches inbuilt to enhance the learning experience as such approaches are proven to help learn languages better in literature. Initially, a prototype spelling game was tested out with 10 participants and received mostly positive feedback. The application was tested out to detect American Sign Language (ASL) hand signs alphabets from A to Z as deaf people would often use letter spelling when referring to words that are not in the official sign languages such as current events or technical topics.

The application leverages the use of computer vision and uses image classification approach to detect hand signs. A Convolutional Neural Networks (CNN) model was built and the built model scored 99.79% accuracy in identifying 29 classes - 26 Alphabets, Space, Delete and Nothing."