Dynamic Metadata Management for Petabyte-Scale File Systems

Appeared in Proceedings of the 2004 ACM/IEEE Conference on Supercomputing (SC '04).

Abstract

In petabyte-scale distributed file systems that decouple read and write from metadata operations, behavior of the metadata server cluster will be critical to overall system performance and scalability. We present a dynamic subtree partitioning and adaptive metadata management system designed to efficiently manage hierarchical metadata workloads that evolve over time. We examine the relative merits of our approach in the context of traditional workload partitioning strategies, and demonstrate the performance, scalability and adaptability advantages in a simulation environment.

Publication date:
November 2004

Authors:
Sage Weil
Kristal Pollack
Scott A. Brandt
Ethan L. Miller

Projects:
Ultra-Large Scale Storage

Available media

Full paper text: PDF

Bibtex entry

@inproceedings{weil-sc04,
  author       = {Sage Weil and Kristal Pollack and Scott A. Brandt and Ethan L. Miller},
  title        = {Dynamic Metadata Management for Petabyte-Scale File Systems},
  booktitle    = {Proceedings of the 2004 ACM/IEEE Conference on Supercomputing (SC '04)},
  month        = nov,
  year         = {2004},
}