Scalable Full-Text Search for Petascale File Systems

Appeared in Proceedings of the 2008 Petascale Data Storage Workshop (PDSW 08).

Abstract

As file system capacities reach the petascale, it is becoming increasingly difficult for users to organize, find, and manage their data. File system search has the potential to greatly improve how users manage and access files. Unfortunately, existing file system search is designed for smaller scale systems, making it difficult for existing solutions to scale to petascale files systems. In this paper, we motivate the importance of file system search in petascale file systems and present a new full-text file system search design for petascale file systems. Unlike existing solutions, our design exploits file system properties. Using a novel index partitioning mechanism that utilizes file system namespace locality, we are able to improve search scalability and performance and we discuss how such a design can potentially improve search security and ranking. We describe how our design can be implemented within the Ceph petascale file system.

Publication date:
November 2008

Authors:
Andrew Leung
Ethan L. Miller

Projects:
Scalable File System Indexing
Ultra-Large Scale Storage

Available media

Full paper text: PDF

Bibtex entry

@inproceedings{leung-pdsw08,
  author       = {Andrew Leung and Ethan L. Miller},
  title        = {Scalable Full-Text Search for Petascale File Systems},
  booktitle    = {Proceedings of the 2008 Petascale Data Storage Workshop (PDSW 08)},
  month        = nov,
  year         = {2008},
}