Loading…
MesosCon18 has ended
Welcome to MesosCon 2018 which will be held in The Village (969 Market St, San Francisco) between November 5th-7th, bringing together users and developers to share and learn about the project and its growing ecosystem.

Tickets are now available to purchase below.


Back To Schedule
Tuesday, November 6 • 3:30pm - 4:10pm
Multi-tenant Spark workflows in Auto Scalable Mesos clusters

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Tweet Share
Recommendation algorithms have been the core of the Netflix product from very early on. Because of their importance, we continually seek to run our machine learning workflows in a reliable, scalable and robust manner.
We will present our design choices on building a Mesos-centric multi-tenant architecture for running Spark-based machine learning workflows that power the algorithms behind Netflix recommendations. Also we will share our experience using the auto-scaling capabilities of Amazon Web Services to dynamically change the size of our clusters to support the allocation of thousands of spark jobs running daily. We will discuss how we are leveraging Apache Spark to deploy batch jobs as well as the interactive use of Zeppelin Notebooks efficiently in this shared environment.
We will cover a few aspects of this multi-tenant platform, such as the Spark scheduler for Mesos, dynamic resource allocation, metrics and dashboards, and Spark history logs.
Speakers
avatar for Pablo Delgado

Pablo Delgado

Machine Learning Engineer, Netflix
Pablo Delgado is a Senior Software Engineer, he works on building infrastructure for machine learning for Personalized Recommendation Algorithms at Netflix. Previously he was working on the recommendation systems stack for personal restaurant recommendations at Opentable. Pablo obtained... Read More →
PD

MesosCon2018 pdf
Tuesday November 6, 2018 3:30pm - 4:10pm PST
A
  Breakout Session