Replication Strategies for Highly Available Peer-to-Peer Storage Systems

Failure is inevitable: disks fail, hosts crash, networks partition, applications stop. Consequently, the principal challenge in designing highly-available systems is to tolerate each failure as it occurs and recover from its effects. For large systems, or systems with unreliable components, such failures can cease to be exceptional events, but instead may become the common case. Perhaps no design point is more challenging in this respect than that faced by heterogeneous peer-to-peer systems. Such systems are typically composed of very large numbers of hosts, of which only a minority may be available at any one time. In this environment, failure is not only common, but pervasive. This paper analyzes the challenges and limitations in building a highly-available storage system in such a peer-to-peer environment. In particular, we explore the design requirements on failure tolerance and failure recovery in environments with limited host availability. Our contributions are threefold: First, we provide an analytic model for reasoning about the efficiency of replication and erasure encoding as temporary storage redundancy mechanisms. Second, we extend this framework to model the availability of groups of files or file systems. Finally, we incorporate the costs of maintaining a given level of availability in the long term by recovering from persistent storage failures. We show that even in environments with pervasive failure it is possible to offer a storage service with a high degree of availability at a moderate cost in storage overhead.