ai-product-evaluation-design

# AI Product Evaluation Design

In the era of LLMs, product development moves from writing static specifications to defining "correctness" through Evals. Since models are stochastic, you cannot "fix a bug" with a single line of code; instead, you must "hill climb" toward better behavior by building robust datasets that measure model performance against your product goals.

## The Three-Tier Evaluation Framework

Depending on the complexity of the feature, use one or more of these evaluation methods:

### 1. Deterministic Evals (Pass/Fail)
Best for extraction, tool-calling, or objective facts.
- **Goal:** Verify the model extracts the exact right data.
- **Example:** If the user says "Remind me to eat at 7 PM," the JSON output for `time` must be `19:00`.
- **Metric:** Accuracy % (Total correct / Total prompts).

### 2. Human Preference Evals (Side-by-Side)
Best for tone, creativity, and visual design (like the "Canvas" layout).
- **Goal:** Compare two model versions (e.g., a baseline vs. a new fine-tuned model).
- **Process:** Present a prompt and two anonymized completions. Ask a human rater: "Which is better for [Specific Goal]?"
- **Metric:** Win Rate (The percentage of time the new model beats the baseline).

### 3. Model-Graded Evals (LLM-as-a-Judge)
Best for scaling quality checks without manual labor.
- **Goal:** Use a high-reasoning model (like o1) to grade the output of a faster, cheaper model.
- **Process:** Give the "Judge" model the rubric of what a "good" response looks like and ask it to score the "Student" model on a scale of 1-5.

## Step-by-Step Process for Designing Evals

### 1. Create the "Ground Truth" Dataset
Build a spreadsheet with the following columns to define the model's target behavior:
- **Input/Prompt:** What the user says (include diverse variations).
- **Baseline Behavior:** How the current model responds.
- **Ideal Behavior:** A hand-written "Golden Response" showing exactly what you want.
- **Rationale:** Why the ideal behavior is better (e.g., "It didn't trigger the UI when it should have stayed in chat").

### 2. Define Decision Boundaries
For agentic features (like Canvas or Task execution), define the "Trigger Boundary":
- **Trigger Scenarios:** Prompt: "Write a 5-page essay." Result: Model opens the document editor.
- **Non-Trigger Scenarios:** Prompt: "Who is the President?" Result: Model stays in the standard chat interface.

### 3. Identify Performance Regressions
When you optimize for one skill (e.g., "Being more concise"), you may accidentally "brain damage" another skill (e.g., "Formatting code correctly").
- Always run your new feature evals alongside a "General Intelligence" eval set to ensure core reasoning hasn't dropped.

## Examples

**Example 1: Deterministic Eval for a "Tasks" Tool**
- **Context:** An AI assistant that sets reminders.
- **Input:** "Remind me to call Mom in two hours." (Sent at 10:00 AM).
- **Expected Output:** `{ "action": "set_reminder", "content": "Call Mom", "time": "12:00" }`.
- **Application:** Run 100 variations of time-based language ("tonight," "in a bit," "next Tuesday") to ensure the extraction logic holds.

**Example 2: Preference Eval for Writing Style**
- **Context:** Improving the "friendly" tone of a document editor.
- **Input:** "Rewrite this paragraph to be more encouraging."
- **Model A:** "You did a good job on the report."
- **Model B:** "This report is a fantastic start! Your analysis of the data is really sharp."
- **Evaluation:** Human rater chooses Model B because it uses specific positive reinforcement instead of generic praise.

## Common Pitfalls

- **Measuring the Wrong Baseline:** Using a weak model as your baseline makes your new model look better than it actually is. Always test against the "state of the art" (SOTA).
- **Neglecting Diversity:** Training or testing only on "happy path" prompts. Include edge cases, slang, and non-English inputs to ensure the model doesn't fail in the wild.
- **The "Over-Refusal" Trap:** Teaching a model to be too safe or helpful can cause it to start refusing valid requests (e.g., the "body paradox" where a model refuses to set an alarm because it "doesn't have a physical body").
- **Ignoring Latency:** A model that is 5% more accurate but 10x slower is often a net-negative for the user experience. Always include "Time to First Token" as an eval metric.