HAGUID: Real-time hand gesture-based approach to UI designing

Abstract

"User Interface (UI) and User Experience (UX) design plays a major role in the development of any application, however the technology used to design have always remained unchanged, stuck on the traditional devices. While those devices may work well, designers may not realize their true capabilities with more advanced technology. Additionally, the traditional devices confine the user to designing using a single hand.

Proposed is a novel UI design tool that applies real-time hand gesture recognition (HGR) to design interfaces. HGR is a popular Computer Vision (CV) field of knowledge that has been applied in many industries. The application of real-time HGR in UI/UX design would cause an immense advancement in efficiency. Moreover, the proposed HGR system aims capture the gesture via a built-in laptop camera or webcam and utilize an innovative hybrid classification technique, CNN-Transformer, that has not been used in an HGR system prior to this. The research aspires to prove the effectiveness of using HGR to design interfaces in order to bridge the identified gap"