GPT-DAE: A Generative Pre-trained Transformer based Approach to Automate Descriptive Answer Evaluation

Abstract

"Essay questions are widely recognised as an effective means of assessing students' comprehension of learning outcomes during examinations. However, the manual evaluation of these essay questions poses challenges, requiring consideration of factors such as required keywords, grammatical accuracy, and overall meaning of the responses. The advent of digitalised examinations has led to the development of various automated grading techniques for descriptive questions. While keyword matching and syntax analysis can be helpful tools, determining the accuracy of a response based solely on meaning presents a significant challenge. This research introduces GPT-DAE, a proposed system for evaluating descriptive questions using a predefined answer script. Leveraging a GPT-based approach, the system identifies semantic relationships within responses to assess the provided answers. During fine-tuning, the author achieved training and validation losses of 0.0000 in one model and the other scored a training loss of 0.2036 and a validation loss of 0.0352. Benchmarking against human-evaluated question scores revealed an accuracy of over 66%, demonstrating the system's potential. However, it is acknowledged that further fine-tuning is essential before deploying the prototype in real-world applications. The study concludes that GPT-DAE has the capability to significantly enhance descriptive answer grading with improved fine-tuning, paving the way for more accurate and efficient evaluation in digitalized examination settings. "