A Real-time Rule-based Audio and Text to Sri Lankan Sign Language Translator Using a 3D Avatar

Abstract

Effective communication is crucial in building relationships, but not all individuals are capable of verbal communication. For instance, individuals with hearing impairments rely on visual language such as Sign Language for communication. This research aims to design, develop and evaluate a real time English audio/text to Sri Lankan Sign Language translator with a 3D interpreter. For translation, the proposed system will contain a novel rule based algorithm that will be using NLP techniques such as dependency parsing for identifying the relationships between words in the given inputs sentences, PoS tagging for identifying named entities such as proper nouns and syntactical reordering to rearrange the tokens in the processed sentence according to the algorithm. According to a usability study conducted, 100% the responses were positive where the respondents found the system to be very effective. Furthermore, all of the implemented avatar sign interpretations were understandable. The proposed system results in a BLEU score of 0.905 and a METEOR score of 0.9472 on performance of the translation modules and results in a WER of 0.19 and a NIST score of 5.17 on speech recognition. For future work, the author will integrate a better speech to text service with automatic sentence detection, the algorithm will be further enhanced to identify over 20 dependency types, a larger corpus will be added to the system and more customization for the existing avatars. A SLSL to English translator module will also be added to facilitate bidirectional communication.