skill-forge-eval

# Skill Evaluation Pipeline

Run structured evaluations against Claude Code skills to verify triggering,
correctness, and quality using a multi-agent pipeline.

## Process

### Step 1: Define Eval Set

Accept eval definitions from:
- **Path to eval set JSON**: `evals/evals.json` or user-specified file
- **Inline prompts**: User provides eval queries directly
- **Auto-generated**: Generate from skill description (see Step 1b)

**Eval set JSON schema:**
```json
{
  "skill_name": "my-skill",
  "skill_path": "./my-skill",
  "evals": [
    {
      "eval_id": 0,
      "eval_name": "descriptive-name",
      "prompt": "The user's task prompt",
      "input_files": [],
      "assertions": [
        {
          "name": "output-has-score",
          "check": "Output contains a numeric score between 0-100",
          "weight": 1.0
        }
      ],
      "should_trigger": true
    }
  ]
}
```

#### Step 1b: Auto-Generate Eval Set

If no eval set exists, generate one:
1. Read the skill's SKILL.md description and instructions
2. Run `python scripts/generate_eval_set.py <skill-path>` to produce a starter set
3. Present the generated set to the user for review and editing
4. User approves or modifies before proceeding

### Step 2: Set Up Workspace

Create the eval workspace **outside** the skill directory to avoid confusing eval
artifacts with skill files. Use a sibling directory or a dedicated location:

```
eval-workspace/
  iteration-1/
    eval-0/
      eval_metadata.json        # Assertions and config for this eval
      with_skill/
        outputs/                # Skill execution outputs
        timing.json             # Token count + duration
        grading.json            # Assertion results + evidence
      baseline/
        outputs/
        timing.json
        grading.json
    eval-1/
      eval_metadata.json
      with_skill/
        outputs/
        timing.json
        grading.json
      baseline/
        outputs/
        timing.json
        grading.json
    benchmark.json              # Aggregated metrics
    benchmark.md                # Human-readable report
```

For each eval directory, create `eval_metadata.json` from the eval set entry:
```json
{
  "eval_id": 0,
  "eval_name": "descriptive-name",
  "prompt": "The user's task prompt",
  "assertions": [...],
  "should_trigger": true
}
```

### Step 3: Execute Eval Runs

For each eval in the set, spawn two parallel runs:

**With-skill run** (delegate to `agents/skill-forge-executor.md`):
```
Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the assertions check>
```

**Baseline run** (delegate to `agents/skill-forge-executor.md`):
- For new skills: run without the skill loaded
- For improved skills: run with the previous version (snapshot it first)

Save timing data to `timing.json` in each run directory:
```json
{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}
```

### Step 4: Grade Results

Delegate to `agents/skill-forge-grader.md` for each completed run:

1. Grade against assertions defined in eval_metadata.json
2. Save results to `grading.json` per run:
```json
{
  "eval_id": 0,
  "run_type": "with_skill",
  "assertions": [
    {
      "name": "output-has-score",
      "passed": true,
      "evidence": "Found score: 87/100 on line 14"
    }
  ],
  "pass_rate": 1.0
}
```

### Step 5: Aggregate and Analyze

1. Run `python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>`
2. This produces `benchmark.json` and `benchmark.md` with:
   - Pass rate per eval (with_skill vs baseline)
   - Average time and token usage
   - Improvement ratio (with_skill / baseline)

3. Delegate to `agents/skill-forge-analyzer.md` to:
   - Surface patterns that aggregate stats might hide
   - Identify consistently failing assertion types
   - Flag regressions from previous iterations

### Step 6: Present Results

Generate a summary report:

```markdown
# Eval Report: [skill-name] — Iteration [N]

## Overall
| Metric | With Skill | Baseline | Delta |
|--------|-----------|----------|-------|
| Pass Rate | X% | Y% | +Z% |
| Avg Time | Xs | Ys | -Zs |
| Avg Tokens | X | Y | -Z |

## Per-Eval Results
| Eval | With Skill | Baseline | Status |
|------|-----------|----------|--------|
| eval-0 | PASS | FAIL | Improved |
| eval-1 | PASS | PASS | Maintained |

## Patterns & Insights
[From analyzer agent]

## Recommendations
[Specific improvements based on failures]
```

### Step 7: Collect Feedback

Save user feedback to `feedback.json`:
```json
{
  "reviews": [
    {
      "run_id": "eval-0-with_skill",
      "feedback": "the chart is missing axis labels",
      "timestamp": "2026-03-06T12:00:00Z"
    }
  ],
  "status": "complete"
}
```

Pass feedback to `/skill-forge evolve` for the next iteration.

## Advanced: Blind Comparison

For rigorous A/B testing between skill versions:
1. Delegate to `agents/skill-forge-comparator.md`
2. Pass two directories: `eval-<ID>/with_skill/outputs/` and `eval-<ID>/baseline/outputs/`
3. Comparator assigns random labels (Version A / Version B) so it cannot know which is new
4. Rates each output on assertion criteria from `eval_metadata.json`
5. Returns preference scores without knowing which is "new" vs "old"

## Error Handling

- **Executor timeout**: If a run exceeds 5 minutes, terminate and mark as `"timed_out": true` in timing.json
- **Executor failure**: If a run crashes, save the error to `error.txt` in the run directory and continue with remaining evals
- **Grading failure**: If grading cannot determine pass/fail, mark assertion as `"passed": null` with evidence explaining why
- **Missing files**: If timing.json or grading.json is missing after a run, flag the eval as incomplete in the report
- **Partial completion**: Always aggregate and report whatever results are available — do not block on one failed eval

## Quality Gates

Before marking an eval run as complete:
- [ ] All evals executed (with_skill + baseline)
- [ ] Timing data captured for every run
- [ ] All assertions graded with evidence
- [ ] Benchmark aggregated with pass rate, time, tokens
- [ ] Analyzer patterns documented
- [ ] Results presented to user