eval-runner

# eval-runner

LLM evaluation framework. Define test suites in YAML, run against model endpoints, score responses, detect regressions.

## When to use

- Testing LLM responses against expected outputs
- Scoring response quality with exact match, regex, or LLM-as-judge
- Comparing model performance (gpt-4o vs gpt-4-turbo, etc.)
- Detecting quality regressions after model or prompt changes
- Running evals in CI/CD pipelines with exit code support

## Quick Start

```bash
# Start the server
eval-runner start

# Run a suite from a YAML file
eval-runner run --suite my-suite.yaml --endpoint openai-prod

# Check for regressions (exits 1 if regression detected)
eval-runner check --run <run-id> --exit-code
```

## YAML Suite Format

```yaml
name: "My Test Suite"
model: "gpt-4o"
endpoint: "openai-prod"

defaults:
  temperature: 0
  max_tokens: 512
  system: "You are a helpful assistant."

cases:
  - id: "case-001"
    name: "Basic factual question"
    input: "What is the capital of France?"
    expected: "Paris"
    scoring:
      - method: contains
        value: "Paris"

  - id: "case-002"
    name: "Quality response check"
    input: "Explain recursion briefly"
    scoring:
      - method: llm-judge
        rubric: "The response clearly explains recursion with an example"
        threshold: 0.8
```

## Scoring Methods

| Method | Description |
|---|---|
| `exact_match` | Response must exactly match expected value (normalized) |
| `contains` | Response must contain the expected substring |
| `regex` | Response must match the regex pattern |
| `json_match` | Parse response as JSON and check a field value |
| `llm-judge` | LLM rates the response against a rubric (0.0-1.0) |

Multiple scorers per case are combined as a weighted average.

## CLI Reference

| Command | Description |
|---|---|
| `eval-runner run --suite <file>` | Run a suite from YAML file |
| `eval-runner run --suite <file> --model <model>` | Override model |
| `eval-runner run --suite <file> --baseline <run-id>` | Compare against baseline |
| `eval-runner check --run <id> --exit-code` | Exit 0/1 based on regression |
| `eval-runner suite list` | List suites stored in server |
| `eval-runner suite import <file>` | Import YAML to server |
| `eval-runner export --run <id> --format json` | Export run results |
| `eval-runner --help` | Show help |

## Environment Variables

| Variable | Description | Default |
|---|---|---|
| `EVAL_PORT` | Server port | 4090 |
| `EVAL_DASHBOARD_PORT` | Dashboard port | 4091 |
| `EVAL_DATA_DIR` | SQLite and data directory | ~/.eval-runner |
| `EVAL_ENCRYPTION_KEY` | 32-byte hex for endpoint key encryption | required |
| `EVAL_CONCURRENCY` | Max parallel case executions per run | 5 |
| `EVAL_CASE_TIMEOUT_MS` | Timeout per case in milliseconds | 30000 |
| `EVAL_REGRESSION_THRESHOLD` | Per-case score drop threshold | -0.1 |
| `EVAL_DEV` | Dev mode (1 or 0) | 0 |

## Regression Detection

A case is regressed if: `current_score - baseline_score < EVAL_REGRESSION_THRESHOLD`

A run is marked as regression if: `regressed_cases / total_cases > regression_rate_threshold` (default 0.05)

Set a baseline when starting a run:
```bash
eval-runner run --suite my-suite.yaml --baseline run_abc123
```

## Troubleshooting

### Endpoint unreachable

Check the base URL in Settings. Test with `curl <base_url>/models`. Verify the API key is saved correctly.

### LLM judge returns null score

The judge model returned malformed JSON. The case scores 0.0. Check the judge model endpoint is working. Try a different judge model.

### Cases timeout

Increase `EVAL_CASE_TIMEOUT_MS`. Long-running cases may need 60,000ms or more.