September 13 -14, 2017 - Los Angeles, CA
Click Here For Information & Registration
Back To Schedule
Friday, September 15 • 11:50am - 12:30pm
Simulating Our Apache Mesos Framework for Fun and Profit and... - Wil Yegelwel, TwoSigma

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

At Two Sigma, we use Cook, our open source batch scheduling mesos framework, to run millions of compute-hours of work for hundreds of users every day. This year, we are working on changing the scheduler optimization algorithm, but we want some confidence that it will actually improve utilization and the user experience (latency). Therefore, we built a mesos simulator which lets us test algorithm changes, without running the whole distributed system, and in doing so, found some subtle bugs and performance bottlenecks, and highlighted areas needing better test coverage.

In this talk, Wil will describe how we built the simulator for cook, some of the challenges in building it, and share the insights we learned by running our historical job traces through it. We will highlight the value and practicality of simulation testing for real, production systems.


Wil Yegelwel

Software Developer, TwoSigma
I'm a developer at Two Sigma Investments working on Distributed System and Analysis tools. Previously, I was a student at Brown University studying Computer Science and Applied Math.

Friday September 15, 2017 11:50am - 12:30pm PDT
Diamond Salon 6