Google Research

From Research to Practice: Experiences Engineering a Production Metadata Database for a Scale Out File System

  • Charles Johnson
  • Kimberly Keeton
  • Charles B. Morrey III
  • Craig A. N. Soules
  • Alistair Veitch
  • Stephen Bacon
  • Oskar Batuner
  • Marcelo Condotta
  • Hamilton Coutinho
  • Patrick J. Doyle
  • Rafael Eichelberger
  • Hugo Kiehl
  • Guilherme Magalhaes
  • James McEvoy
  • Padmanabhan Nagarajan
  • Patrick Osborne
  • Joaquim Souza
  • Andy Sparkes
  • Mike Spitzer
  • Sebastien Tandel
  • Lincoln Thomas
  • Sebastian Zangaro
Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST 2014), USENIX

Abstract

HP’s StoreAll with Express Query is a scalable commercial file archiving product that offers sophisticated file metadata management and search capabilities. A new REST API enables fast, efficient searching to find all files that meet a given set of metadata criteria and the ability to tag files with custom metadata fields. The product brings together two significant systems: a scale out file system and a metadata database based on LazyBase. In designing and building the combined product, we identified several real-world issues in using a pipelined database system in a distributed environment, and overcame several interesting design challenges that were not contemplated by the original research prototype. This paper highlights our experiences.

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work