AI Query Compilation for Unified and Optimized Execution

In this vision paper, we propose a novel architectural paradigm for accelerated AI query execution via a unified compiled execution strategy. By compiling the hybrid \textit{AI Query} as a whole—integrating both standard SQL relational constructs and LLM inference layers into a single, unified tensor compute graph—we completely eliminate PCIe data movement bottlenecks across execution boundaries and enable global compiler optimizations. We demonstrate the viability of this unified execution paradigm on synthetic benchmarks and select AI queries from \textit{SemBench}, achieving up to 5.3x latency speedup and 9.8x throughput speedup on TPUs, and outline a research roadmap of open technical challenges to realize this vision.