SchemaRAG: Semantic dictionary based Agentic LLM for Text-to-SQL

Abstract

Text-to-SQL is an efficient way to generate SQL queries and retrieve data from databases using human natural language. This approach helps non-technical users in different domains such as education, healthcare, and finance to access information from databases for different use cases such as decision making, reporting and analytics without needing technical expertise to write SQL queries. With the rapid development of Large Language models over recent years, an increase in Text-to-SQL studies can be observed and the performance of these models has greatly improved due to advanced reasoning capabilities of LLMs. However, due to the inherent ambiguity of the natural language and the complexity of the schema and required SQL queries, the existing Text-to-SQL solutions suffer from issues such as hallucinations, lack of domain knowledge and the inability to accurately generate complex queries with join operations. To address these limitations, an Agentic LLM system for Text-to-SQL tasks was implemented during this study with the use of Retrieval-Augmented Generation (RAG) for domain knowledge enhancement of the LLM. Four LLM agents were utilized in the Text-to-SQL workflow. The process starts with an Assistant agent capturing user’s natural language questions and its role is to delegate schema retrieval, query generation and query validation tasks to the Retrieval Agent, Query Generation Agent and Validation Agent respectively, in the same order. The schema knowledge is stored in a database as vector embeddings, which are used to fetch relevant schema information related to the user’s natural language question based on the semantic similarities. SchemaRAG achieved an execution accuracy of (EX) 73.9% during the testing phase. The proposed system performs reasonably well considering the minimal effort needed for SQL query generation compared to existing work, which is evident through evaluation feedback. The proposed system shows great potential in real world use cases with its usability improvements such as minimal reliance on user interaction and enabling access to non-technical users through automatic schema knowledge retrieval