Cooper: Expedite Batch Data Dissemination in Computer Clusters with Coded Permutation Gossips

  • Author / Creator
    Liu, Yan
  • Data transfers happen frequently in server clusters for software and application deployment, and in parallel computing clusters to transmit parameters in batches among servers between computation stages. This thesis presents Cooper, an optimized prototype system to speedup multi-batch data transfers among a cluster of servers, leveraging a theoretically proven optimal algorithm called "permutation gossip" which employs randomly permuted node connections to best utilize bandwidth and random linear code to maximize the useful information transmitted. By chunking the file into a proper number of blocks, we present a pipelining technique to parallelize the coding operation and network transfer on the process level, realizing the theoretically promised benefits of random linear codes. More importantly, for batch-based or multiple transfers, we propose priority-based scheduling algorithms to overlap the transfers of different batches, which further reduce the transfer finish time of each batch, while only delaying the first batch for a constant time. We present an asynchronous and distributed prototype implementation of Cooper and deploy it on Amazon EC2 for evaluation. Based on results from real experiments, we show that Cooper can significantly speedup data transfers and reduce redundant transmissions in server clusters as compared to state-of-the-art content distribution tools, including BitTorrent and an optimized random-block transfer strategy based on buffer negotiation in a wide range of practical settings.

  • Subjects / Keywords
  • Graduation date
  • Type of Item
  • Degree
    Master of Science
  • DOI
  • License
    This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.
  • Language
  • Institution
    University of Alberta
  • Degree level
  • Department
    • Department of Electrical and Computer Engineering
  • Specialization
    • computer engineering
  • Supervisor / co-supervisor and their department(s)
    • Niu, Di (Department of Electrical and Computer Engineering)
  • Examining committee members and their departments
    • Khabbazian, Majid (Department of Electrical and Computer Engineering)
    • Ardakani, Masoud (Department of Electrical and Computer Engineering)