Trace Analysis of Large Scale Storage Systems

Published as MS project report, University of California, Santa Cruz.

Abstract

Storage systems for scientific and industrial workloads involve working with petabytes of data. These systems often have complex hierarchies of different types of storage media through which data movement takes place. It is important to understand the behavior of such a system, including migration, replication, per user read/write patterns, per task usage, as well as trends over longer periods of time, such as a month or a year. Such analysis will help us identify the system usage, reduce redundant reads and writes. Trace analysis also allows us to identify and differentiate between recurring tasks, related tasks, and schedule them with necessary priorities, to improve throughput and reduce latency.
We present our uniform trace analysis framework, which is designed to take in traces across multiple large scale systems, and compare the behavior of the archives over time. The system can take in data across multiple formats, and present a 1:1 comparison of attributes as well as usage across systems. We present an analysis of the CERN EOS filesystem traces, traces gathered from CERN’s production system over a year. The analysis is across 2.49 billion unique events that happened on the EOS filesystem. We plan to integrate this trace analysis with traces from other scientific labs and archives, to compare and contrast behavior of large scale storage systems.

Publication date:
June 2018

Authors:
Devashish Purandare

Projects:
Archival Storage
Designing systems for QLC flash
Tracing and Benchmarking
Ultra-Large Scale Storage

Available media

Full paper text: PDF

Bibtex entry

@unpublished{purandare-masters,
  author       = {Devashish Purandare},
  title        = {Trace Analysis of Large Scale Storage Systems},
  howpublished = {MS project report, University of California, Santa Cruz},
  month        = jun,
  year         = {2018},
}