Managing Google’s data lake: an overview of the Goods system

For most large enterprises today, data constitutes their core assets, along with code and infrastructure.
Indeed, for most enterprises, the amount of data that they produce internally has exploded. At the same
time, in many cases, engineers and data scientists do not use centralized data-management systems
and and up creating what became known as a data lake—a collection of datasets that often are not
well organized or not organized at all and where one needs to “fish” for the useful datasets. In this
paper, we describe our experience building and deploying Goods, a system to manage Google’s internal
data lake. Goods crawls Google’s internal infrastructure and builds a catalog of discovered datasets,
including structured files, databases, spreadsheets, or even services that provide access to the data.
Goods extracts metadata about datasets in a post-hoc way: engineers continue to generate and organize
datasets in the same way as they have before, and Goods provides values as without disrupting teams’
practices. The technical challenges that we had to address resulted both from the scale and heterogeneity
of Google’s data lake and our decision to extract metadata in a post-hoc manner. We believe that many
of the lessons that we learned are applicable to building large-scale enterprise-level data-management
systems in general.

• Data Management