atft-pipeline

# ATFT Pipeline Skill

## Mission
- Provision fresh or historical parquet datasets for ATFT-GAT-FAN with GPU-accelerated ETL.
- Maintain deterministic feature graphs (approx. 395 engineered factors, 307 active).
- Guard J-Quants API quota, credential sanity, and cache health to prevent training stalls.

## When To Engage
- Any request mentioning dataset builds, ETL, J-Quants, cache, RAPIDS/cuDF, or feature graph refresh.
- Pre-training sanity checks (“ensure latest dataset”, “verify cache integrity”).
- Recovery tasks (“resume interrupted dataset job”, “clean corrupted cache shards”).

## Preflight Checklist
- Confirm `nvidia-smi` reports at least one free A100 80GB GPU; fallback to CPU only if GPU unavailable.
- Validate credentials: `.env` contains `JQUANTS_AUTH_EMAIL/PASSWORD` and `JQUANTS_PLAN_TIER`.
- Ensure `python -m pip install -e .` already executed (dependencies + entry points).
- Check latest health snapshot: `tools/project-health-check.sh --section dataset`.
- Inspect existing dataset for reuse: `ls -lh output/ml_dataset_latest_full.parquet`.

## Core Playbooks

### 1. Background Five-Year Refresh (default)
1. `make dataset-check-strict` — GPU + secrets verification.
2. `make dataset-bg START=<optional> END=<optional>` — SSH-safe background run with logging in `_logs/dataset`.
3. `tail -f _logs/dataset/*.log` — monitor progress (auto prints PID + PGID).
4. `make cache-stats` — ensure cache hit-rate & size in expected bounds (<2.5 TB).
5. `python scripts/pipelines/run_full_dataset.py --dry-run` — confirm metadata integrity without rebuild.

### 2. Hotfix / Forced Refresh
1. `make dataset-gpu-refresh START=YYYY-MM-DD END=YYYY-MM-DD` — bypasses cached parquet + API throttle aware.
2. `make datasets-prune` — keep latest dataset generation only.
3. `make cache-prune CACHE_TTL_DAYS=90` — evict stale graph shards to recover disk.

### 3. Resource-Constrained Fallback
1. `make dataset-check` — relaxed diagnostics (CPU acceptable).
2. `make dataset-cpu START=YYYY-MM-DD END=YYYY-MM-DD` — chunked Pandas path.
3. `make dataset-safe-resume` — resume from last safe checkpoint if memory pressure triggered fallback.

### 4. Graph Feature Investigation
1. `python scripts/pipelines/run_full_dataset.py --inspect-graph --start YYYY-MM-DD --end YYYY-MM-DD`.
2. `python -c "import polars as pl; df = pl.read_parquet('output/ml_dataset_latest_full.parquet'); print(df.select(pl.all().is_null().sum()))"` — null audit.
3. `make cache-monitor` — per-window edge density + overlap stats.

## Observability Hooks
- `_logs/dataset/` for job logs, `cache/*.json` metadata for cache.
- `ml_dataset_latest_full_metadata.json` for column coverage & horizon alignment.
- `benchmark_output/dataset_timestamps.json` to confirm pipeline duration vs baseline (target: <42m GPU path).

## Failure Triage
- **Credential errors** → run `python scripts/pipelines/run_full_dataset.py --auth-test`.
- **CUDA OOM** → rerun with `make dataset-safe` (40GB RMM pool pre-configured).
- **API rate limits** → throttle via `make dataset-gpu REFRESH_THROTTLE=1`.
- **Corrupted parquet** → `make dataset-rebuild` then `python tools/parquet_validator.py output/ml_dataset_latest_full.parquet`.

## Codex Collaboration
- Escalate complex ETL debugging or architectural refactors via `./tools/codex.sh "Diagnose dataset pipeline bottleneck"` (leverages OpenAI Codex deep reasoning).
- For long-running autonomous maintenance, schedule `./tools/codex.sh --max --exec "Perform full dataset pipeline audit"` off-hours (uses `.mcp.json` from the Codex repo for filesystem/git context).
- When Codex proposes changes, sync learnings back here and refresh dataset runbooks if any commands or defaults shift.

## Handoff Notes
- Always update `dataset_features_detail.json` if schema changes.
- Announce new dataset snapshot in `EXPERIMENT_STATUS.md` with generation timestamp and settings.
- Surface anomalies (missing tickers, new features) via `docs/data_quality/` reports.