Continuous Pipelines at Google

This article focuses on the real life challenges of managing data processing pipelines of depth and complexity. It considers the frequency continuum between periodic pipelines that run very infrequently through continuous pipelines that never stop running, and discusses the discontinuities that can produce significant operational problems. A fresh take on the master­slave model is presented as a more reliable and better scaling alternative to the periodic pipeline for processing Big Data.

• Software Engineering

• Distributed Systems and Parallel Computing