Ian gives a practice run of his qualifying exam.

The past two decades have seen an explosion in the growth and role of long-term digital archival storage. While the traditional role of tertiary storage as an archive has persisted, there are many new use-cases as well, such as public historical document archives and climate sensor data. Yet, despite this expansion, our understanding of long-term storage is out of date. We have no insights into how these new archival use-cases behave, and even our understanding of tertiary storage behavior is decades old. Without up to date information on their behavior we cannot validate the effectiveness of both current and future archival architectures.

To this end, we propose examining several archival systems across a variety of use-cases to bring our knowledge of the field up to date. To explore tertiary storage we have obtained datasets and will be analyzing logs detailing file system behavior of super-compute archives from both Los Alamos National Laboratory and the National Center for Atmospheric Research. To explore some of the new archival use-cases we obtained access logs from two publicly accessible archives maintained by the California Department of Water Resources and the Washington State Digital Archives. They store water sensor data and historical documents respectively. By examining these datasets, our analysis will provide insight into how these relatively new use-case behave in a real-world environment.

The final portion of our proposal seeks to address how one effectively monitors and traces a large, distributed system over long periods of time. One of the most vexing problems we consistently encountered in our analyses were myopic and disconnected logs and traces, making understanding aggregate system behavior difficult, if not impossible at times. Adding to the challenge is the necessity of years of logs often being need to understand long-term behavior, as well as the increasingly distributed nature of modern systems. To explore this issue we are proposing Janus, a framework for long-term system monitoring. Janus will provide a scalable and reliable framework to enable reliable storage of trace data as it is created, periodic transformation of logs to reduce space overhead, and user-specified summarization and tagging of important events as they occur.