Overview

Cosmos by Astronomer is an open‑source library that seamlessly bridges dbt Core and Apache Airflow, enabling data teams to convert dbt projects directly into orchestrated Airflow DAGs or Task Groups. With just a few lines of code, Cosmos auto-generates tasks that mirror dbt models, inheriting Airflow’s robust features such as scheduling, retries, SLA alerts, datasets, and detailed observability. It translates Airflow connections into dbt credentials dynamically—no separate profiles.yml needed—and isolates dependencies by running dbt in dedicated Python virtual environments, eliminating conflicts. On Astronomer’s managed platform (Astro), Cosmos enhances pipelines further by capturing open-lineage metadata automatically, deploying via CLI, and offering Docker-based isolation and rich alerting.

Features and Capabilities

• Effortless DAG/task generation: Use DbtDag or DbtTaskGroup to convert dbt projects into Airflow workflows with under 10 lines of code.
• Model-to-task mapping & observability: Each dbt model becomes its own Airflow task, complete with lineage, retries, SLA, and SLA-triggered alerts.
• Dynamic credential mapping: Maps Airflow connections to dbt profiles at runtime—no manual profile YAML, and compatible with secret backends.
• Dependency isolation: Executes dbt within custom Python virtual environments to prevent version conflicts with Airflow.
• Auto-run dbt tests: Cosmos triggers dbt tests post-model materialization, ensuring data quality without extra configuration.
• Diverse execution modes: Supports local, Kubernetes, AWS EKS, Azure Container Instances, and container-based execution for flexible deployments.
• Doc generation & hosting within Airflow: Utilizes pre-built operators (DbtDocsOperator, DbtDocsS3/GCS/Azure) to generate and expose dbt docs via Airflow’s UI.
• Remote manifest support: Allows remote execution by reading dbt project manifests from S3, GCS, or Azure Blob, optimizing CI workflows.
• Astro-enhanced lineage & deployment: On Astronomer’s Astro platform, Cosmos automatically captures lineage metadata, supports CLI-based DAG deployments, and centralizes alerting and dependency isolation via Docker.