airflow

Apache Airflow workflow orchestration. Use for data pipelines.

7 stars

byG1Joshi

View on GitHub Installation ↓

Best use case

airflow is best used when you need a repeatable AI agent workflow instead of a one-off prompt.

Apache Airflow workflow orchestration. Use for data pipelines.

Teams using airflow should expect a more consistent output, faster repeated execution, less prompt rewriting.

When to use this skill

You want a reusable workflow that can be run more than once with consistent structure.

When not to use this skill

You only need a quick one-off answer and do not need a reusable workflow.
You cannot install or maintain the underlying files, dependencies, or repository context.

Installation

Claude Code / Cursor / Codex

$curl -o ~/.claude/skills/airflow/SKILL.md --create-dirs "https://raw.githubusercontent.com/G1Joshi/Agent-Skills/main/skills/ai-ml/airflow/SKILL.md"

Manual Installation

Download SKILL.md from GitHub
Place it in .claude/skills/airflow/SKILL.md inside your project
Restart your AI agent — it will auto-discover the skill

How airflow Compares

Feature / Agent	airflow	Standard Approach
Platform Support	Not specified	Limited / Varies
Context Awareness	High	Baseline
Installation Complexity	Unknown	N/A

Frequently Asked Questions

What does this skill do?

Apache Airflow workflow orchestration. Use for data pipelines.

Where can I find the source code?

You can find the source code on GitHub using the link provided at the top of the page.

SKILL.md Source

# Airflow

Apache Airflow is the standard for data engineering pipelines. v3.0 (2025) introduces **Event-driven Triggers** and a modern React UI.

## When to Use

- **ETL/ELT**: Scheduling nightly data warehouse loads.
- **ML Ops**: Retraining models when new data arrives.
- **Dependency Management**: "Run Task B only if Task A succeeds".

## Core Concepts

### DAGs (Directed Acyclic Graphs)

Defined in Python.

### Task SDK

New in v3.0. Allows writing tasks in any language, not just Python.

### Edge Executor

Run tasks on remote edge devices.

## Best Practices (2025)

**Do**:

- **Use the TaskFlow API**: `@task` decorators are cleaner than `PythonOperator`.
- **Use Datasets**: Define data-aware scheduling (`schedule=[Dataset("s3://bucket/file")]`).

**Don't**:

- **Don't put top-level code in DAG files**: It runs every scheduler heartbeat.

## References

- [Airflow Documentation](https://airflow.apache.org/)