A Novel Framework to Improve Retrieval Augmented Generation in Medical Chatbot with Iterative Follow-up Questions

Abstract

In the not-too-distant past, people have started using healthcare chatbots in an effort to get instant responses to their questions concerning their health. Most of the health chatbots still falter when patients ask them complex questions or don't have some necessary information. Most of the systems respond to the original question and neglect requesting extra questions. To that end, the responses they make are not comprehensive or not entirely accurate. That is a failing, more particularly in the area of healthcare, when one should establish the needs of the user before a suggestion is given. In a bid to counter that problem, I have developed a new class of chatbot that asks follow up questions before even attempting the solution. I trained a language model into asking these follow up questions and paired it with a RAG (Retrieval-Augmented Generation) architecture. The chatbot asks the users follow up questions and asks answers from authoritative sources like medical APIs and PDFs. I used tools like ChromaDB for caching medical articles and Ollama to run the language model locally. I even had a scoring system that screened out the best potential answers. Everything else was married with Streamlit with an interactive UI such that the users enjoy a seamless chatting experience. I experimented with the validation of the system using a collection of information retrieval metrics such as Precision@k, MRR, and nDCG in testing the answer quality. I attempted to make a few people test the system such as medical or technical students and provide feedback regarding the same. Most of them reported that the chatbot answered with more information compared to the other systems that involved follow-up questioning. Overall, the project proved that follow-up questioning enables the improvement of the chatbot and answering with more specific and informative responses.