Jump to Content

From Research to Practice: Experiences Engineering a Production Metadata Database for a Scale Out File System

Charles Johnson
Kimberly Keeton
Charles B. Morrey III
Craig A. N. Soules
Alistair Veitch
Stephen Bacon
Oskar Batuner
Marcelo Condotta
Hamilton Coutinho
Patrick J. Doyle
Rafael Eichelberger
Hugo Kiehl
Guilherme Magalhaes
James McEvoy
Padmanabhan Nagarajan
Patrick Osborne
Joaquim Souza
Andy Sparkes
Mike Spitzer
Sebastien Tandel
Lincoln Thomas
Sebastian Zangaro
Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST 2014), USENIX


HP’s StoreAll with Express Query is a scalable commercial file archiving product that offers sophisticated file metadata management and search capabilities. A new REST API enables fast, efficient searching to find all files that meet a given set of metadata criteria and the ability to tag files with custom metadata fields. The product brings together two significant systems: a scale out file system and a metadata database based on LazyBase. In designing and building the combined product, we identified several real-world issues in using a pipelined database system in a distributed environment, and overcame several interesting design challenges that were not contemplated by the original research prototype. This paper highlights our experiences.