VoiceBlockly: Voice Code Generation in Block-Based Programming Using a Novel Multi-Agent Framework

Abstract

Block-based programming (BBP) has proven effective in teaching programming concepts, offering a visual and more intuitive approach than traditional text-based programming. However, accessibility in BBP remains an underexplored area, particularly for students with disabilities who may benefit from alternative input methods, such as voice commands. This project aims to bridge this accessibility gap by integrating natural language processing (NLP) capabilities into a BBP interface, enabling users to generate programming blocks through voice commands. To achieve effective voice to block code generation, the project employs a novel multi-agent (MA) framework that integrates several large language models (LLM) working together. Each model is designed to interpret user commands and generate corresponding block syntax. A novel aggregation algorithm based on uncertainty quantification (UQ) is introduced to combine outputs from multiple agents, ensuring the accuracy and validity of the generated blocks. This approach enables the system to produce high quality output by leveraging the collective strengths of multiple agents. The prototype demonstrates that the BBP environment effectively translates NL input into accurately rendered code blocks within the interface. Testing and evaluation results indicate that the proposed MA framework outperforms the leading state-of-the-art (SOTA) MA framework by 6% in programming tasks, achieving a Pass@1 score of 75% on the HumanEval dataset. Furthermore, it surpasses the same MA framework by 6% in general domain tasks, attaining an accuracy of 65% on the TruthfulQA dataset.