Managing Google’s data lake: an overview of the Goods system

For most large enterprises today, data constitutes their core assets, along with code and infrastructure. Indeed, for most enterprises, the amount of data that they produce internally has exploded. At the same time, in many cases, engineers and data scientists do not use centralized data-management systems and and up creating what became known as a data lake—a collection of datasets that often are not well organized or not organized at all and where one needs to “fish” for the useful datasets. In this paper, we describe our experience building and deploying Goods, a system to manage Google’s internal data lake. Goods crawls Google’s internal infrastructure and builds a catalog of discovered datasets, including structured files, databases, spreadsheets, or even services that provide access to the data. Goods extracts metadata about datasets in a post-hoc way: engineers continue to generate and organize datasets in the same way as they have before, and Goods provides values as without disrupting teams’ practices. The technical challenges that we had to address resulted both from the scale and heterogeneity of Google’s data lake and our decision to extract metadata in a post-hoc manner. We believe that many of the lessons that we learned are applicable to building large-scale enterprise-level data-management systems in general.

• Data Management