llm-judge

# llm-judge

Use an LLM model to score responses against a natural language rubric. Returns a 0.0-1.0 score with reasoning.

## How it works

The judge model receives:
1. The rubric describing what a good response looks like
2. The actual response to evaluate

The judge returns a JSON object: `{"score": 0.0-1.0, "reason": "brief explanation"}`

## Configuration in YAML

```yaml
scoring:
  - method: llm-judge
    rubric: "The answer clearly explains the return policy and includes the timeframe"
    threshold: 0.8       # score >= 0.8 = pass (default: 0.7)
    weight: 2            # double weight vs other scorers (default: 1)
    endpoint: openai-prod  # override default judge endpoint
    model: gpt-4o-mini    # override default judge model
```

## Rubric Writing Tips

Write rubrics as positive statements of what a good response contains:

Good rubric:
```
"The response mentions the 30-day return window and explains the condition requirement"
```

Bad rubric (vague):
```
"The response is good"
```

Good rubric for tone:
```
"The response is empathetic, acknowledges the customer's frustration, and provides a concrete next step"
```

## Default Configuration

Set globally in Settings or environment:
- Default judge endpoint: `openai-prod` (configurable)
- Default judge model: `gpt-4o-mini` (configurable - cheaper models work well for judging)

## Prompt Injection Protection

Case input is wrapped in XML delimiters before insertion into the judge prompt to prevent injection attacks:

```
<case_input>
{user's case input}
</case_input>
```

## Cost Considerations

llm-judge makes one additional LLM call per scored case. For 100 test cases, each with one judge scorer, you incur 100 extra LLM calls. Use `gpt-4o-mini` or `claude-3-haiku` as the judge to minimize cost.