Running Apache Spark on Mesos Timothy Chen tim@mesosphere.io © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. About me: - Distributed Systems Architect @ Mesosphere - Lead Containerization engineering - Apache Mesos, Drill PMC / Committer - Maintain Apache Spark Mesos Schedulers © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. mesos.berkeley.edu/mesos_ tech_report.pdf 6 © 2015 Mesosphere, Inc. “Our results show that Mesos can achieve nearoptimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) 7 © 2015 Mesosphere, Inc. mesos.berkeley.edu/mesos_ validate our hypothesis tech_report.pdf “To ..., we have also built a new framework on top of Mesos called Spark...” 8 © 2015 Mesosphere, Inc. Apache Mesos © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. Mesos: level of indirection scheduler scheduler Mesos (master) Mesos (agents) © 2015 Mesosphere, Inc. Mesos Improve utilization by sharing cluster Support multiple frameworks with weighted DRF and roles Allow Isolation among frameworks and jobs Simplified Operations and Development Mesos Community Frameworks & Tools © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. Spark Cluster Abstraction Standalone Mesos YARN © 2015 Mesosphere, Inc. © 2015 Mesosphere, Inc. Mesos Master © 2015 Mesosphere, Inc.