MesosCon North America 2017

Distributed deep learning is essential to speed up complex model training, scale out to hundreds of GPUs, and shard models that can not be fit into a single machine. With recent advance on deep learning models in self-driving car areas such as lane-detection, perception and so on, it is important to enable distributed deep learning with large-scale GPU clusters.

This presentation will discuss our design and implementation of running distributed TensorFlow on top of Mesos clusters with hundreds of GPUs. It leverages several key features offered by Mesos such as GPU isolation and nested containers. We also implement several features in our scheduler to support GPU and Gang scheduling, task discovery and dynamic port allocation. Finally, we will show the speed up of distributed training on Mesos using an example TensorFlow model for image classification.