PRISM: Navigating Cost–Accuracy Trade-offs for NL2SQL
Abstract
Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) tasks, but their effectiveness depends heavily on system configuration. Real-world deployments involve tuning a complex pipeline of components, including LLM choice, prompting strategy, schema linking, and example selection, while navigating trade-offs between accuracy and monetary cost. Critically, the optimal configuration varies across databases, yet prior work lacks a principled framework for such database-aware optimization.
We present PRISM, a two-phase system that systematically identifies high-accuracy, cost-efficient NL2SQL configurations tailored to each database. Phase 1 applies Bayesian Optimization with a cost-aware acquisition function to efficiently explore the configuration space and select a diverse set of high-performing single-candidate pipelines. Phase 2 ensembles these configurations to further improve execution accuracy. Experiments on the BIRD benchmark demonstrate that PRISM achieves 69.48\% execution accuracy in the single candidate setting, improving accuracy by 2.34\% and reducing cost by 92\% compared to baselines. In the ensemble setting, PRISM boosts accuracy further to 75.3\%.
We present PRISM, a two-phase system that systematically identifies high-accuracy, cost-efficient NL2SQL configurations tailored to each database. Phase 1 applies Bayesian Optimization with a cost-aware acquisition function to efficiently explore the configuration space and select a diverse set of high-performing single-candidate pipelines. Phase 2 ensembles these configurations to further improve execution accuracy. Experiments on the BIRD benchmark demonstrate that PRISM achieves 69.48\% execution accuracy in the single candidate setting, improving accuracy by 2.34\% and reducing cost by 92\% compared to baselines. In the ensemble setting, PRISM boosts accuracy further to 75.3\%.
