Table Format Optimizations in Managed Spark for Dataproc
Abstract
Efficient data lifecycle management is critical for modern LakeHouse architectures, yet standard open source implementations of transactional updates and storage maintenance often suffer from prohibitive I/O and compute costs. This paper presents novel, format-aware optimizations implemented within the Dataproc Spark engine, including workload aware optimizations. For transactional workloads on Lakehouse, we introduce Precision Merge, which utilizes a row-level tracking mechanism to minimize shuffle and wasted compute. Furthermore, we propose significant enhancements to table maintenance operations, utilizing Direct Binary Merge for efficient compaction and a multi-tiered, asynchronous framework for VACUUM processes. Extensive benchmarking demonstrates that these combined strategies deliver over a 50% reduction in execution time, a 10x decrease in shuffle volume, and up to a 10x speedup in routine maintenance, providing a highly scalable and cost-effective solution for high-throughput Lakehouse management.
