Enhancing Domain-Specific Accuracy and Reliability in Large Language Models: A Case Study on a Marriage and Divorce Information Chat System Based on Sri Lankan Law.

Abstract

"The effectiveness of Large Language Models (LLMs) is critical for providing legal advice about Sri Lankan marriage and divorce laws. The novel improvements using Retrieval-Augmented Generation (RAG) and QLORA-efficient fine-tuning techniques are investigated in this thesis. Using a customised dataset, the QLORA-efficient fine-tuning method modifies the Mistral-7B-Instruct-v0.2 LLM, utilising quantification and Low-Rank Adaptation approaches to maximise accuracy. Simultaneously, the addition of verifiable legal text from genuine court papers enhances model replies through RAG integration, improving accuracy and decreasing false positives. Comprehensive metrics evaluation, such as total scores, BERTScore F1, and ROUGE scores, show that the RAG-enhanced model regularly outperforms the fine-tuned model and the original LLM. The effectiveness of the RAG methodology in providing relevant and well-supported legal advice is further validated by expert reviews. A chatbot that made use of the RAG model was then created to operationalize this better strategy. This application not only increases the model's faithfulness but also creates a trustworthy resource for anyone looking for accurate legal advice regarding marriage and divorce laws in Sri Lanka. "