MesosCon18

One of the strengths of Mesos is the ability to simultaneously run diverse schedulers. We recently had a use case that required running thousands of instances of different frameworks (Marathon, Cassandra, Spark, Jenkins) on a single cluster. While it is well understood that Mesos clusters can scale to tens of thousands of agents, running thousands of frameworks on them is still uncharted territory.

We set out to do scale tests to explore this territory. In this talk, we will present the treasures that we found and dragons encountered along the way.

We will describe the tooling we developed to monitor and execute the tests, the challenges we faced on the allocator such as offer starvation and fragmentation, the inevitable performance problems and surprising behaviours.

We will share our learnings, the things things we fixed, and some of the best practices that we recommend.

Apache Mesos and Apache Aurora enable engineering teams at Twitter to run production services at scale, without the pain associated with traditional management of bare-metal hardware. Within Ads Serving, we face the interesting challenge of running (deploying, operating, staffing oncall) production systems that contain code from many teams. Others depend on us for shipping their feature, and we carry the pager for their code in addition to ours.

Historically, we have invested in release testing tooling to help provide signal on the viability of a future deployment. While this was helpful, it ultimately had gaps in coverage that resulted in failed canaries and delayed deployments. One example is that our ML based ads serving system uses a feedback control mechanism to trade off ad quality for service availability. This introduces additional challenges to validate the health of a change list or a release candidate.

In this talk, we will discuss the multi-faceted problems we were facing, our design for tackling these challenges, and the results we have observed since going live. We will cover Aurora-based solutions for automated load-testing of code reviews (each diff, before merge), release candidate load testing (multiple diffs in a deployment), canary analysis, and deployment of multiple logical clusters across multiple datacenters … all with two clicks :)

Applications on a shared host are competing on resources access. Network resources are particularly sensitive to such usage contention, and if not managed properly can lead to great performance degradation for applications. It is very easy for one noisy neighbor to impact the observed response time of other applications.

At Criteo, we are running some network intensive business critical applications on our Mesos clusters.

We chose to develop a custom solution to offer our end users both network isolation and network bandwidth as a first class resource handled by Mesos and its schedulers.

This talk is going to present our journey so far, our approach to add a new resources offered to all users of our frameworks, a deep dive into our solution, and the challenges we faced, offering a return on experience on both Mesos ecosystem development and operations.

With the increasing scale of Uber’s business, efficient use of cluster resources is important to reduce the cost per trip. As we have learned when operating Mesos clusters in production, it is a challenge to overcommit resources for latency-sensitive services due to their large spread of resource usage patterns. Uber also has significant demand on running large-scale batch jobs for marketplace intelligence, fraud detection, maps, self-driving vehicles etc.

In this talk, we will present Peloton, a Unified Resource Scheduler for collocating heterogeneous workloads in shared Mesos clusters. The goal of Peloton is to manage compute resources more efficiently while providing hierarchical max-min fairness guarantees for different teams. Peloton schedules large-scale batch jobs with millions of tasks and also supports distributed TensorFlow jobs with thousands of GPUs.

Recently, Mesosphere added support for Multi-Kubernetes on Mesos. This enables both Kubernetes and Mesos workloads to run side-by-side on the same cluster. Not only that, different Kubernetes clusters can run on the same Mesos cluster. This talk will focus on the different network design alternatives to achieve this, challenges and pitfalls with each and finally provides details of the approach chosen. The talk takes references from the implementation in DC/OS, however, the approach is generic enough to be applicable on other platforms on top of Mesos.

Docker container is a foundational building block of Uber infrastructure, but distributing the docker images to all hosts in a system is a scaling problem we have been facing at Uber for some time.

We started with a single docker registry in 2015, gradually added replicas and cache layers. However as Uber grew larger and more complex, the number of hosts and containers also grew exponentially; Also as the deployment system become more automated, rebalance and rollback happens more frequently. Docker registry couldn’t catch up with the number of container being deployed, which grew 10x every year to millions of docker pull requests per day. Eventually we decided to take a P2P distribution approach to tackle this problem - the project is called Kraken.

Kraken is a P2P docker image distribution system. It’s loosely based on BitTorrent protocol, fully compatible with docker registry API, and supports pluggable storage backends like S3, HDFS, etc. It successfully solved scaling problems we saw under different scenarios, also greatly sped up container deployment.

This talk will cover:
- Docker registry scaling challenges
- Architecture of Kraken
- Design and optimizations decisions we took
- Production performance numbers

Statically provisioning and re-configuring dedicated storage resources has proven to be a maintenance burden when running co-located tasks of heterogeneous frameworks atop a single Mesos cluster. Users have long desired a combination of flexible, plug-and-play storage options and self-provisioning frameworks that leverage the resource accounting native to Mesos clusters.

Since its v1.5 release, Mesos has supported the Container Storage Interface (CSI) industry standard that aims to simplify storage driver integration for operators of platforms like Mesos and others, as well as for storage vendors. Resource Provider (RP) components form the backbone of the vendor resource supply chain within Mesos. They allow for both the proper accounting, and life cycle management, of resources whose control plane is implemented by external code: the CSI plugin. By leveraging recent additions to the Mesos APIs it is now possible to write frameworks that dynamically self-provision "mount"- and "block"-type disk resources that are fit for the the task at hand.

This talk provides an overview of the new storage components and primitives that have shipped in recent Mesos releases, as well as sketches an outline of a self-provisioning framework that utilizes the newly available APIs.

Yelp has been using Apache Mesos to power its production clusters for four years, and these clusters run at significant scale. We began to wonder how much further we could scale our clusters before Mesos bottlenecked our growth, so we performed a number of load tests to attempt to understand these limits. In this talk, we discuss the results of these load tests; specifically,

* how well the Mesos master APIs perform under heavy load, particularly with a large number of frameworks present
* how well the Mesos containerizer (aka the Universal Container Runtime, or UCR) performs compared to the Docker containerizer
* how the Mesos UCR performs when launching a group of containers, or "pods", using the Container Networking Interface to allocate an IP-per-container

All of our load test results (and the load testing harness) will be made available to the community for further analysis.

Details of a production tested secret distribution mechanism for Mesos that is framework agnostic. A run down of the pros and cons of other ways of distributing secrets on Mesos A discussion on reliability and how we can make sure that our tasks keep running if our secret store goes down Discussion about the Mesos modules architecture and ideas for how to make it better

At Uber, we manage a Mesos fleet of tens of thousands of hosts, and as we scale up to span multiple datacenters and cloud platforms, we've developed a system called CLM (Cluster Lifecycle Manager) to automatically manage host maintenance, cluster operations, and infra upgrades without impacting running services' SLAs. CLM serves as a missing layer between service orchestration and host / resource management. By using an extensible system to gather issues on our hosts from multiple sources, such as hardware failures or misconfigurations, we are able to repair or remove those hosts before they impact production. We use goalstate config to specify the expected state of clusters and automatically converge on that, while integrating with the orchestration layer to ensure we operate safely without causing any disruption to service health.

Mesos has its own container runtime (aka. UCR) and companies have been using it for years in production. Based on the pure Linux Kernel namespaces and cgroups, Mesos supports different container image formats with advantages of extensible container storage, networking, and security. As of today it supports all major container standards, such as CSI, CNI and OCI soon.

In the past one year, there are a lot of accomplishments in Mesos Containerization. For instances, Linux ambient capabilities, File-based secret with the new secret resolver module, Cgroup Blkio Isolation, Automatic Container Image GC, Standalone Container, Windows Support improvement, Persistent Volume Resizing, and many other fantastic new features. In this talk, we will review the achievements in Containerization and discuss the future of Mesos Containerization.

This talk covers the last year of work porting Mesos to Windows, particularly with improvements to stout, libprocess, the containerizer, and the inclusion of libarchive (for Linux too!). We'll be talking about challenges we faced and solutions we found, as well as what the Windows port needs next.

Monitoring the workload on a Mesos cluster is a challenge for which no industry standard exists. There are several distinct problems to solve, specific to Mesos.

Firstly, the sheer scale of possible metrics. Mesos enables and encourages microservice architectures. More services means more metrics - and a scale issue.

Secondly, metric identity resolution. This means understanding relationship between metrics from nodes, frameworks, executors, tasks and containers.

Lastly, cardinality. We must tag metrics with metadata for context, but each combination of label and tags increases the cardinality of the dataset.

We will outline each problem, round up the state of metrics with regard to orchestrated environments, and present a reference solution using Telegraf and Prometheus. All tools (and our own libraries) are free, open source and will be available immediately for use with Mesos.

Recommendation algorithms have been the core of the Netflix product from very early on. Because of their importance, we continually seek to run our machine learning workflows in a reliable, scalable and robust manner.
We will present our design choices on building a Mesos-centric multi-tenant architecture for running Spark-based machine learning workflows that power the algorithms behind Netflix recommendations. Also we will share our experience using the auto-scaling capabilities of Amazon Web Services to dynamically change the size of our clusters to support the allocation of thousands of spark jobs running daily. We will discuss how we are leveraging Apache Spark to deploy batch jobs as well as the interactive use of Zeppelin Notebooks efficiently in this shared environment.
We will cover a few aspects of this multi-tenant platform, such as the Spark scheduler for Mesos, dynamic resource allocation, metrics and dashboards, and Spark history logs.

The Command Line Interface (CLI) of Apache Mesos has recently been rewritten in Python 3. Aside from the refactoring, the CLI offers new features for developers and operators. In this talk, we will see:
- How to configure the CLI to work with your cluster.
- How to use the main CLI commands, `mesos task attach` and `mesos task exec`.
- How to extend the CLI by developing your own plugins.
After this talk, you will be capable of monitoring your clusters using this new interface and extending this component to make it work with your workflow.