Herald Kllapi
Authored Publications
Sort By
Preview abstract
Distributed federated SQL query engines frequently
materialize query results into files in data lakes. Optimizing the
sizes of these files (e.g., balancing their sizes) is crucial for the
efficiency of not only the materialization queries themselves, but
also subsequent queries that read these files. Existing techniques
to manage file size include specifying output targets (e.g., number
of partitions), using cardinality estimation for materialized data
volume, performing full shuffles prior to materialization to obtain
accurate statistics, and/or post (background) compactions to
merge small files. All of these solutions have limitations in
practice; they typically do not provide any strong guarantees or
can be prohibitively expensive when they do (e.g., they require
background compactions to merge small files, which requires
additional I/O and CPU cost). Most of the existing techniques
can produce many small files during materialization and are
sensitive to data skew.
In this paper, we propose a novel method that streamlines
materialization within the same query execution and provides
guarantees on the file sizes using statistics at runtime. Our
approach does not require a full shuffle before the materialization
operator in order to get accurate statistics, making it attractive
and robust in practice. To show the effectiveness of our approach,
we present production metrics from F1 Query at Google that has
been running this functionality for the majority of its production
workload over many quarters. Our approach reduces the number
of files produced by a factor of 100 or more, and as a result,
it avoids creating tens of billions of files per day, saving storage
and computation cost.
View details
