Astronomer is a data engineering platform that collects, processes and unifies users' data so that they can get straight to analytics and data science. Initially, Astronomer built their pipeline out of AWS-managed services. But, when they ran into the limitations of these services, they realized that they had committed to proprietary components, which prevented them from changing cloud providers.
In this talk, Astronomer outlines their transition to open source services running on Apache Mesos and DC/OS. They started by using Apache Airflow instead of AWS Simple Workflow Service and went on to replace their Kinesis streams with Apache Kafka, and deploy their stream processing applications with Marathon. They will review their remaining challenges and the solutions they are considering. It took some time and effort to for Astronomer decouple from AWS, but they are glad they did.
