MesosCon18

One of the strengths of Mesos is the ability to simultaneously run diverse schedulers. We recently had a use case that required running thousands of instances of different frameworks (Marathon, Cassandra, Spark, Jenkins) on a single cluster. While it is well understood that Mesos clusters can scale to tens of thousands of agents, running thousands of frameworks on them is still uncharted territory.

We set out to do scale tests to explore this territory. In this talk, we will present the treasures that we found and dragons encountered along the way.

We will describe the tooling we developed to monitor and execute the tests, the challenges we faced on the allocator such as offer starvation and fragmentation, the inevitable performance problems and surprising behaviours.

We will share our learnings, the things things we fixed, and some of the best practices that we recommend.