The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators

SQL has been augmented with AI operators, enabling modern data analytics platforms to drive insights from both structured and unstructured data. We observe that while current Text-to-SQL systems can successfully generate these AI-augmented queries using in-context examples, the real challenge is evaluation. Traditional metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of embedded LLMs. In this paper, we formalize these unique evaluation failure modes and introduce a Multilayered Evaluation Framework that decouples deterministic relational database logic from flexible AI operator semantics. Using SemBench, a multimodal benchmark for SQL with AI operators, we demonstrate that traditional Execution Accuracy severely penalizes valid queries (achieving only a 36% detection rate for correct translations), even though our generator produces 70% production-viable queries. Furthermore, even state-of-the-art LLM-based auto-raters falsely reject 24% of valid queries due to the complexity of evaluating hybrid logic simultaneously. In contrast, our framework achieves a 100% detection rate on correct queries and a 91% success rate in identifying flawed translations, establishing a reliable standard for benchmarking AI-powered SQL generators.