The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
•
4 likes•2,052 views
The document summarizes Milind Bhandarkar's work developing Hamster, a system for running MPI applications on Hadoop YARN. Some key points: - Hamster allows MPI applications to run alongside Hadoop dataflow jobs on the same cluster managed by YARN. It implements an MPI runtime on top of YARN. - Hamster's design leverages OpenMPI's strengths while allowing it to integrate with YARN. It includes an application master, node service, and scheduler component. - Performance tests show Hamster has low overhead and scales well for large MPI jobs. It introduces only a small performance penalty compared to running MPI natively with OpenMPI. - Example results are shown
![About Me
• http://www.linkedin.com/in/milindb
• Founding member of Hadoop team atYahoo! [2005-2010]
• Contributor to Apache Hadoop since v0.1
• Built and led Grid SolutionsTeam atYahoo! [2007-2010]
• Parallel Programming Paradigms [1989-today] (PhD cs.illinois.edu)
• Center for Development of Advanced Computing (C-DAC), National Center
for Supercomputing Applications (NCSA), Center for Simulation of Advanced
Rockets, Siebel Systems (acquired by Oracle), Pathscale Inc. (acquired by
QLogic),Yahoo!, LinkedIn, and Pivotal (formerly Greenplum)](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
Similar to The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster (20)
Recently uploaded
Recently uploaded (20)
The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
- 1. The Zoo Expands Labrador 💛 Elephant,Thanks to Hamster Milind Bhandarkar Chief Scientist, Pivotal Software, Inc.
- 2. About Me • http://www.linkedin.com/in/milindb • Founding member of Hadoop team atYahoo! [2005-2010] • Contributor to Apache Hadoop since v0.1 • Built and led Grid SolutionsTeam atYahoo! [2007-2010] • Parallel Programming Paradigms [1989-today] (PhD cs.illinois.edu) • Center for Development of Advanced Computing (C-DAC), National Center for Supercomputing Applications (NCSA), Center for Simulation of Advanced Rockets, Siebel Systems (acquired by Oracle), Pathscale Inc. (acquired by QLogic),Yahoo!, LinkedIn, and Pivotal (formerly Greenplum)
- 3. Hamster • Hadoop and MPI on the same cluster • Runtime for OpenMPI applications onYARN • Available on Pivotal HD
- 4. Why MPI ? •Hadoop Dataflow paradigms (MapReduce, TeZ etc) not suitable for iterative applications •Message Passing Interface (MPI) •Mature standard •Used extensively in HPC •Huge ecosystem
- 5. MPI in Science & Engg Earth Atmosphere Chemistry Biology Math Nuclear
- 6. MPI in Industry Mechanical ar Finance/bankOil Exploration Cryptography Spacecraft
- 7. OpenMPI •Mature Open Source implementation of MPI 3.0 Standard (mpi-forum.org) •New BSD license •30+ contributing organizations from academia, research and industry •http://open-mpi.org
- 9. Pluggable
- 10. Hamster Design •YARN as Resource Manager •Hamster Application Manager •Manages MPI jobs •(tries to) Implement Gang-Scheduling •Leverages OMPI/ORTE strengths •Wire-up,Task monitoring, Fast Interconnect
- 11. Hamster Architecture Resource Manager Scheduler AMService Node Manager Node Manager Node Manager … Proc/Container Framework Daemon NSMPI Scheduler HNP MPI AM Proc/Container …RM-AM AM-NM RM-NodeManager Client Client-RM Aux Srvcs Proc/Container Framework Daemon NS Proc/Container … Aux Srvcs RM- NodeManager
- 12. Hamster AppMaster • Master daemon for MPI ( similar to JobTracker in MapReduce) • Implements and participates in theYARN-RM App lifecycle protocol • Maintains heartbeat with RM to ensure liveness • MPI Scheduler - Negotiates resource allocation with YARN-RM • Head Node Process (HNP) - manages job execution
- 13. Hamster Node Service •User-level daemon per MPI job •Manages task execution •Coarse-grained container management •Bootstrapped byYARN-NM •Implemented asYARN Auxiliary Service
- 15. Why GraphLab on Hadoop ? •Graph Analytics & Machine Learning only one stage in E2E data pipeline •ETL/Preprocessing •Building Graphs from fact & dimension tables •Publishing analytics results, post-processing
- 16. GraphLab 2.2 •Communication patterns based on Data •SeveralToolkits (Graph Analytics + ML Algorithms) available •Graph-Programming API •Uses MPI for communication
- 17. Pivotal HD HDFS HBase Pig, Hive, Mahout Map Reduce Sqoop Flume Resource Management & Workflow Yarn Zookeeper Apache Pivotal Command Center Configure, Deploy, Monitor, Manage Spring XD Pivotal HD Enterprise Spring Xtension Framework Catalog Services Query Optimizer Dynamic Pipelining ANSI SQL + Analytics HAWQ – Advanced Database Services Distributed In-memory Store Query Transactions Ingestion Processing Hadoop Driver – Parallel with Compaction ANSI SQL + In-Memory GemFire XD – Real-Time Database Services MADlib Algorithms Oozie Virtual Extensions Graphlab, Open MPI
- 18. Performance
- 19. Test Environment •Pivotal Analytics Workbench Cluster •Pivotal HD 1.1 (Apache Hadoop 2.0.5) •Hamster - 1.0, OpenMPI-1.7.2 •515 nodes •2x6-core Westmere, 48GB RAM, 12x2TB SATA, Mellanox FDR Infiniband
- 20. Null Job •Measures overhead of launching MPI jobs •Tests scalability of resource allocation, launching and wire-up •Sub-linear scalability (slightly worse than O(logN) •Overhead of launching 15000 processes = 1 minute
- 21. Total RuntimeTime(Sec.) 5 18.75 32.5 46.25 60 Process number 0 4000 8000 12000 16000 E2E time
- 22. AllocationTimeTime(Sec.) 1 2.25 3.5 4.75 6 Number of Processes 0 4000 8000 12000 16000 Allocation Time
- 23. LaunchTimeTime(Sec.) 0 7.5 15 22.5 30 Number of processes 0 4000 8000 12000 16000 Launch Time
- 24. Comparison with OpenMPI •HPL (HP Linpack forTop-500) •Number of processes 50—1000 •Hamster 1% slower than OpenMPI
- 25. HPL - Hamster vs OpenMPI Time(Sec.) 0 30 60 90 120 1000 500 200 50
- 26. GraphLab ALS •Wikipedia dataset •4.3 M terms, 3.3M documents, 513M occurrences •17 Processes •5 Iterations
- 28. GraphLab PageRank •Twitter Dataset •4.1 M nodes, 1.4 B edges •Data Size : 26GB •NP = 17 •50 iterations: 297 seconds •100 iterations: 339 seconds
- 29. Questions?