eval-audit

# /eval-audit

> Audit an LLM evaluation pipeline for correctness and reliability.

## Purpose

Systematically audit an LLM evaluation pipeline across 6 diagnostic areas: data quality, metric validity, judge reliability, coverage gaps, statistical rigor, and pipeline integrity. Each area produces structured Check/Finding pairs with severity ratings. The audit catches common eval pitfalls — label leakage, metric gaming, distribution mismatch, underpowered samples, and judge bias — before they corrupt decision-making. Outputs prioritized recommendations and suggests follow-up skills.

## Usage

```bash
# Full audit of an eval pipeline
/eval-audit --pipeline evals/

# Audit specific diagnostic area
/eval-audit --pipeline evals/ --area judge-reliability

# Audit with custom severity threshold
/eval-audit --pipeline evals/ --min-severity major

# Audit from eval config file
/eval-audit --config evals/config.yaml

# Quick scan (skip deep statistical checks)
/eval-audit --pipeline evals/ --quick
```

## Arguments

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--pipeline` | string | required | Path to evaluation pipeline directory |
| `--config` | string | — | Path to eval config file (alternative to `--pipeline`) |
| `--area` | enum | `all` | Diagnostic area: `data`, `metrics`, `judges`, `coverage`, `statistics`, `integrity`, `all` |
| `--min-severity` | enum | `minor` | Minimum severity to report: `critical`, `major`, `minor` |
| `--quick` | flag | false | Quick scan (skip computationally expensive checks) |
| `--output` | string | stdout | Write report to file |
| `--format` | enum | `markdown` | Output format: `markdown`, `json` |

## Workflow

1. **Inventory** — Scan the pipeline directory. Identify eval datasets, metric definitions, judge prompts, scoring scripts, and result files. Build a pipeline map.
2. **Data Quality** — Check eval datasets for: label balance, duplicate entries, data leakage from training, stale or outdated examples, ambiguous ground truth, insufficient diversity.
3. **Metric Validity** — Check metrics for: alignment with actual goals (Goodhart's Law), gaming susceptibility, sensitivity to edge cases, proper aggregation (micro vs macro), confidence intervals.
4. **Judge Reliability** — Check LLM judges for: position bias, verbosity bias, self-preference, prompt sensitivity, inter-rater agreement, calibration against human labels.
5. **Coverage Gaps** — Check for: untested capabilities, missing edge cases, distribution mismatch between eval and production, capability dimensions not represented.
6. **Statistical Rigor** — Check for: sufficient sample sizes, proper significance testing, multiple comparison correction, effect size reporting, reproducibility.
7. **Pipeline Integrity** — Check for: deterministic execution, version pinning, result caching correctness, data pipeline bugs, reporting accuracy.
8. **Prioritize** — Rank all findings by severity and impact. Group related findings. Recommend next skills to address issues.

## Examples

### Full pipeline audit
```
/eval-audit --pipeline evals/summarization/

## Eval Audit — evals/summarization/

### Area 1: Data Quality
| Check | Finding | Severity |
|-------|---------|----------|
| Label balance | 73% positive, 27% negative — skewed | MAJOR |
| Duplicates | 12 duplicate entries found (4.8%) | MINOR |
| Staleness | 40% of examples from pre-2024 data | MAJOR |
| Leakage | No leakage detected | PASS |

### Area 3: Judge Reliability
| Check | Finding | Severity |
|-------|---------|----------|
| Position bias | Judge prefers response A 61% of the time | CRITICAL |
| Verbosity bias | Longer responses scored 0.8 points higher on average | MAJOR |
| Human agreement | Cohen's kappa = 0.42 (moderate) — below 0.6 threshold | MAJOR |

### Recommendations
1. [CRITICAL] Fix position bias — use `/validate-evaluator` to calibrate
2. [MAJOR] Rebalance dataset — use `/synthetic-data` to generate minority class
3. [MAJOR] Update stale examples — 40% of eval data predates current model behavior
```

## Output

```markdown
## Eval Audit Report

### Pipeline: <path>
### Date: <date>
### Overall Health: RED | YELLOW | GREEN

### Findings Summary
- Critical: N
- Major: N
- Minor: N
- Passing checks: N

### [Diagnostic Area Sections with Check/Finding tables...]

### Recommended Next Skills
1. `/validate-evaluator` — Calibrate judge against human labels
2. `/synthetic-data` — Generate balanced eval data
3. `/error-analysis` — Deep-dive on failure patterns

### Priority Actions
1. [Action + owner + deadline suggestion]
2. ...
```

## Dependencies

- Eval pipeline files (datasets, prompts, scripts, results)
- `/validate-evaluator` — Recommended follow-up for judge issues
- `/synthetic-data` — Recommended follow-up for data gaps
- `/error-analysis` — Recommended follow-up for failure patterns
- Statistical libraries for significance testing (when not in `--quick` mode)