Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
September 13 -14, 2017 - Los Angeles, CA
Click Here For Information & Registration
View analytic
Friday, September 15 • 11:50am - 12:30pm
Simulating Our Apache Mesos Framework for Fun and Profit and... - Wil Yegelwel, TwoSigma

Sign up or log in to save this to your schedule and see who's attending!

At Two Sigma, we use Cook, our open source batch scheduling mesos framework, to run millions of compute-hours of work for hundreds of users every day. This year, we are working on changing the scheduler optimization algorithm, but we want some confidence that it will actually improve utilization and the user experience (latency). Therefore, we built a mesos simulator which lets us test algorithm changes, without running the whole distributed system, and in doing so, found some subtle bugs and performance bottlenecks, and highlighted areas needing better test coverage.

In this talk, Wil will describe how we built the simulator for cook, some of the challenges in building it, and share the insights we learned by running our historical job traces through it. We will highlight the value and practicality of simulation testing for real, production systems.

Speakers
WY

Wil Yegelwel

Software Developer, TwoSigma
I'm a developer at Two Sigma Investments working on Distributed System and Analysis tools. Previously, I was a student at Brown University studying Computer Science and Applied Math.


Friday September 15, 2017 11:50am - 12:30pm
Diamond Salon 6

Attendees (17)