MesosCon North America 2017

Managing resource utilization is one of the hardest aspects of operating Twitter’s Mesos clusters. As the number of services grows and their resource shapes diversify, the bin packing problem becomes increasingly difficult. Tuning for optimal performance would reduce resource usage, and ease the bin packing burden. However, the multitude of available knobs, heterogeneous hardware, the large number of services, and software and hardware upgrades together make the tuning problem combinatorially intractable.

At Twitter we are developing a system that continuously performs automated tuning of services running in our Mesos clusters, using a machine learning technique called Bayesian optimization. This technique allows us to efficiently search very large parameter spaces to optimize specific performance metrics. We describe our system and share initial results.