Overview

Soda Pipeline Testing enables teams to embed data quality tests early in development and production workflows. It uses SodaCL—a human-readable check language—to define and run data-as-code quality validations within CI/CD pipelines, such as in Airflow, Dagster, Databricks or GitHub Actions. The core platform supports built-in metrics, version-controlled checks, anomaly detection, threshold validations, and integrates seamlessly with Soda Cloud for dashboards, alerts, and collaboration. It helps catch faulty data far upstream, preventing downstream impacts on analytics, reporting, and ML models.

Features and Capabilities

• Core Functionality: Write and version control data quality checks as code (SodaCL) executable in pipelines (CI/CD, Airflow, Databricks, Dagster).
• Built‑in Checks & Metrics: Preconfigured metrics like missing values, duplicates, freshness, schema validation, numeric distribution, invalid formats, and row counts—scalable across datasets.
• Shift‑left Testing: Run quality scans early—during development or pre-commit in GitHub Actions—to detect issues before production merge.
• Multi‑Platform Integration: Compatible with orchestration engines (Airflow, Dagster, Azure Data Factory, Databricks), transformation tools (dbt), and data catalogs (Alation, Atlan, Metaphor).
• Anomaly Detection & Observability: Enables metric-based anomaly dashboards, dynamic thresholds, and alerting via platform or via Soda Cloud integrations.
• Soda Cloud Collaboration: Centralized dashboards, alert routing to Slack, Jira, MS Teams, incident tracking, and no-code/low-code data contracts to align data producers and consumers.
• Deployment Flexibility: Supports self-operated CLI library, self-hosted or Soda-hosted agent, as well as programmatic SDK usage.
• Compliance & Industry Support: Designed to automate data integrity for regulated industries—including financial services—supporting compliance frameworks such as BCBS 239 for risk reporting and automated auditing trails.