ct-grade

# Session Grading Guide

Session grading evaluates agent behavioral patterns against the CLEO protocol. It reads the audit log for a completed session and applies a 5-dimension rubric to produce a score (0-100), letter grade (A-F), and diagnostic flags.

## When to Use Grade Mode

Use grading when you need to:
- Evaluate how well an agent followed CLEO protocol during a session
- Identify behavioral anti-patterns (skipped discovery, missing session.end, etc.)
- Track improvement over time across multiple sessions
- Validate that orchestrated subagents followed protocol

Grading requires audit data. Sessions must be started with the `--grade` flag to enable audit log capture.

## Starting a Grade Session

### CLI

```bash
# Start a session with grading enabled
ct session start --scope epic:T001 --name "Feature work" --grade

# The --grade flag enables detailed audit logging
# All CLI operations are recorded for later analysis
```

## Running Scenarios

The grading rubric evaluates 5 behavioral scenarios that map to protocol compliance:

### 1. Fresh Discovery
Tests whether the agent checks existing sessions and tasks before starting work. Evaluates `session.list` and `tasks.find` calls at session start.

### 2. Task Hygiene
Tests whether task creation follows protocol: descriptions provided, parent existence verified before subtask creation, no duplicate tasks.

### 3. Error Recovery
Tests whether the agent handles errors correctly: follows up `E_NOT_FOUND` with recovery lookups (`tasks.find`), avoids duplicate creates after failures.

### 4. Full Lifecycle
Tests session discipline end-to-end: session listed before task ops, session properly ended, CLI usage patterns.

### 5. Multi-Domain Analysis
Tests progressive disclosure: use of `admin.help` or skill lookups, use of progressive disclosure for programmatic access.

## Evaluating Results

### CLI

```bash
# Grade a specific session
ct grade <sessionId>

# List all past grade results
ct grade --list
```

## Understanding the 5 Dimensions

Each dimension scores 0-20 points, totaling 0-100.

### S1: Session Discipline (20 pts)

| Points | Criteria |
|--------|----------|
| 10 | `session.list` called before first task operation |
| 10 | `session.end` called when work is complete |

**What it measures**: Does the agent check existing sessions before starting, and properly close sessions when done?

### S2: Discovery Efficiency (20 pts)

| Points | Criteria |
|--------|----------|
| 0-15 | `find:list` ratio >= 80% earns full 15; scales linearly below |
| 5 | `tasks.show` used for detail retrieval |

**What it measures**: Does the agent prefer `tasks.find` (low context cost) over `tasks.list` (high context cost) for discovery?

### S3: Task Hygiene (20 pts)

Starts at 20 and deducts for violations:

| Deduction | Violation |
|-----------|-----------|
| -5 each | `tasks.add` without a description |
| -3 | Subtasks created without `tasks.find {exact:true}` parent check |

**What it measures**: Does the agent create well-formed tasks with descriptions and verify parents before creating subtasks?

### S4: Error Protocol (20 pts)

Starts at 20 and deducts for violations:

| Deduction | Violation |
|-----------|-----------|
| -5 each | `E_NOT_FOUND` error not followed by recovery lookup within 5 ops |
| -5 | Duplicate task creates detected (same title in session) |

**What it measures**: Does the agent recover gracefully from errors and avoid creating duplicate tasks?

### S5: Progressive Disclosure Use (20 pts)

| Points | Criteria |
|--------|----------|
| 10 | `admin.help` or skill lookup calls made |
| 10 | Progressive disclosure used for programmatic access |

**What it measures**: Does the agent use progressive disclosure (help/skills) for efficient protocol access?

## Interpreting Scores

### Letter Grades

| Grade | Score Range | Meaning |
|-------|-----------|---------|
| **A** | 90-100 | Excellent protocol adherence. Agent follows all best practices. |
| **B** | 75-89 | Good. Minor gaps in one or two dimensions. |
| **C** | 60-74 | Acceptable. Several protocol violations need attention. |
| **D** | 45-59 | Below expectations. Significant anti-patterns present. |
| **F** | 0-44 | Failing. Major protocol violations across multiple dimensions. |

### Reading the Output

The grade result includes:
- **score/maxScore**: Raw numeric score (e.g., `85/100`)
- **percent**: Percentage score
- **grade**: Letter grade (A-F)
- **dimensions**: Per-dimension breakdown with score, max, and evidence
- **flags**: Specific violations or improvement suggestions
- **entryCount**: Number of audit entries analyzed

### Flags

Flags are actionable diagnostic messages. Each flag identifies a specific behavioral issue:

- `session.list never called` -- Check existing sessions before starting new ones
- `session.end never called` -- Always end sessions when done
- `tasks.list used Nx` -- Prefer `tasks.find` for discovery
- `tasks.add without description` -- Always provide task descriptions
- `Subtasks created without parent existence check` -- Verify parent exists first
- `E_NOT_FOUND not followed by recovery lookup` -- Follow errors with `tasks.find`
- `No admin.help or skill lookup calls` -- Load `ct-cleo` for protocol guidance
- `No progressive disclosure calls` -- Use `admin.help` or skill lookups

## Common Anti-patterns

| Anti-pattern | Impact | Fix |
|-------------|--------|-----|
| Skipping `session.list` at start | -10 S1 | Always check existing sessions first |
| Forgetting `session.end` | -10 S1 | End sessions when work is complete |
| Using `tasks.list` instead of `tasks.find` | -up to 15 S2 | Use `find` for discovery, `list` only for known parent children |
| Creating tasks without descriptions | -5 each S3 | Always provide a description with `tasks.add` |
| Ignoring `E_NOT_FOUND` errors | -5 each S4 | Follow up with `tasks.find` or `tasks.exists` |
| Creating duplicate tasks | -5 S4 | Check for existing tasks before creating new ones |
| Never using `admin.help` | -10 S5 | Use progressive disclosure for protocol guidance |
| No progressive disclosure calls | -10 S5 | Use `admin.help` or skill lookups for protocol guidance |

## Grade Result Schema

Grade results are stored in `.cleo/metrics/GRADES.jsonl` as append-only JSONL. Each entry conforms to `schemas/grade.schema.json` with these fields:

- `sessionId` (string, required) -- Session that was graded
- `taskId` (string, optional) -- Associated task ID
- `totalScore` (number, 0-100) -- Aggregate score
- `maxScore` (number, default 100) -- Maximum possible score
- `dimensions` (object) -- Per-dimension `{ score, max, evidence[] }`
- `flags` (string[]) -- Specific violations or suggestions
- `timestamp` (ISO 8601) -- When the grade was computed
- `entryCount` (number) -- Audit entries analyzed
- `evaluator` (`auto` | `manual`) -- How the grade was computed

## CLI Grade Operations

| Command | Description |
|---------|-------------|
| `ct grade <sessionId>` | Grade a specific session |
| `ct grade --list` | List past grade results |