Hadoop branching: Architectural impacts on energy and performance

• Author(s) / Creator(s)

  • Polato, I.
  • Barbosa, D.
  • Hindle, Abram
  • Kon, F.

• Data centers are notorious energy consumers. In fact, studies have shown that for every $1 spent on hardware in the datacenter, $0.50 is spent on powering this hardware over its lifetime. Data centers host real or virtual (i.e., cloud) clusters that often execute large compute jobs using MapReduce, of which Hadoop is a popular implementation. Like other successful open source projects, Hadoop has been maintained and evolved over time with new resource management features being added over time in an effort to improve performance, raising questions as to whether such architectural evolution has achieved its goal, and if so, at what cost. In this work we apply Green Mining to find out that later versions of Hadoop — who exhibit more dynamic resource control — can suffer from serious energy consumption performance regressions.

• Date created

  2015

• Subjects / Keywords

  • Hadoop branching
  • virtual clusters
  • MapReduce
  • resource management features
  • performance improvement
  • architectural evolution
  • green mining
  • dynamic resource control
  • energy consumption performance regressions

• Type of Item

  Conference/Workshop Presentation

• DOI

  https://doi.org/10.7939/r3-4spf-j582

• License

  Attribution 4.0 International