Long Context Evaluation For NL2SQL
Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities across a range of natural language processing tasks. In
particular, improvements in reasoning abilities and the expansion
of context windows have opened new avenues for leveraging these
powerful models. NL2SQL is challenging in that the natural language question is inherently ambiguous, while the SQL generation
requires a precise understanding of complex data schema and semantics. One approach to this semantic ambiguous problem is to
provide more and sufficient contextual information.
In this work, we explore the performance and the latency tradeoffs of the extended context window (a.k.a., long context) offered
by Google’s state-of-the-art LLM (gemini-1.5-pro). We study the
impact of various contextual information, including column example values, question and SQL query pairs, user-provided hints, SQL
documentation, and schema. To the best of our knowledge, this
is the first work to study how the extended context window and
extra contextual information can help NL2SQL generation with
respect to both accuracy and latency cost. We show that long context LLMs are robust and do not get lost in the extended contextual
information. Additionally, our long-context NL2SQL pipeline based
on Google’s Gemini-pro-1.5 achieves strong performance across
multiple benchmark datasets without fine-tuning or expensive
self-consistency based techniques.
particular, improvements in reasoning abilities and the expansion
of context windows have opened new avenues for leveraging these
powerful models. NL2SQL is challenging in that the natural language question is inherently ambiguous, while the SQL generation
requires a precise understanding of complex data schema and semantics. One approach to this semantic ambiguous problem is to
provide more and sufficient contextual information.
In this work, we explore the performance and the latency tradeoffs of the extended context window (a.k.a., long context) offered
by Google’s state-of-the-art LLM (gemini-1.5-pro). We study the
impact of various contextual information, including column example values, question and SQL query pairs, user-provided hints, SQL
documentation, and schema. To the best of our knowledge, this
is the first work to study how the extended context window and
extra contextual information can help NL2SQL generation with
respect to both accuracy and latency cost. We show that long context LLMs are robust and do not get lost in the extended contextual
information. Additionally, our long-context NL2SQL pipeline based
on Google’s Gemini-pro-1.5 achieves strong performance across
multiple benchmark datasets without fine-tuning or expensive
self-consistency based techniques.
