- 171 views
- 155 downloads
Cooper: Expedite Batch Data Dissemination in Computer Clusters with Coded Permutation Gossips
-
- Author / Creator
- Liu, Yan
-
Data transfers happen frequently in server clusters for software and application deployment, and in parallel computing clusters to transmit parameters in batches among servers between computation stages. This thesis presents Cooper, an optimized prototype system to speedup multi-batch data transfers among a cluster of servers, leveraging a theoretically proven optimal algorithm called "permutation gossip" which employs randomly permuted node connections to best utilize bandwidth and random linear code to maximize the useful information transmitted. By chunking the file into a proper number of blocks, we present a pipelining technique to parallelize the coding operation and network transfer on the process level, realizing the theoretically promised benefits of random linear codes. More importantly, for batch-based or multiple transfers, we propose priority-based scheduling algorithms to overlap the transfers of different batches,
which further reduce the transfer finish time of each batch, while only delaying the first batch for a constant time. We present an asynchronous and distributed prototype implementation of Cooper and deploy it on Amazon EC2 for evaluation. Based on results from real experiments, we show that Cooper can significantly speedup data transfers and reduce redundant transmissions in server clusters as compared to state-of-the-art content distribution tools, including BitTorrent and an optimized random-block transfer strategy based on buffer negotiation in a wide range of practical settings. -
- Subjects / Keywords
-
- Graduation date
- Spring 2015
-
- Type of Item
- Thesis
-
- Degree
- Master of Science
-
- License
- This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.