Orchestrating Data Pipelines with Dagster

As data applications grow in complexity and heterogeneity, many platform teams find that managing the interdependencies between components becomes harder and harder. Dagster, the data orchestrator, is a programming model and associated operational tools designed to support data applications through the entire software engineering cycle, from local dev and test, through deployment in production, operational monitoring, debugging and extension.

We’ll discuss some of the ways different teams have used the core Dagster abstractions and toling to tighten their dev and test cycles, ensure correctness and reliability in their data processing pipelines, deploy to disparate environments, integrate with dozens of other ecosystem tools to empower non-engineer personas, and solve operational issues in their data applications.

Max Gasner
Software Engineer, Elementl
LinkedIn: https://www.linkedin.com/in/max-gasner-63333022/