MesosCon18

At Uber, we manage a Mesos fleet of tens of thousands of hosts, and as we scale up to span multiple datacenters and cloud platforms, we've developed a system called CLM (Cluster Lifecycle Manager) to automatically manage host maintenance, cluster operations, and infra upgrades without impacting running services' SLAs. CLM serves as a missing layer between service orchestration and host / resource management. By using an extensible system to gather issues on our hosts from multiple sources, such as hardware failures or misconfigurations, we are able to repair or remove those hosts before they impact production. We use goalstate config to specify the expected state of clusters and automatically converge on that, while integrating with the orchestration layer to ensure we operate safely without causing any disruption to service health.