uat-execute

# UAT Execute

Execute a UAT plan against live MCP connections. Runs each phase sequentially, tracks pass/fail per test, files issues for failures, and always runs cleanup.

## Usage

```bash
# Execute a UAT plan
/uat-execute .aiwg/testing/uat/plan-gitea-20260227.md

# Dry run — show what would be executed without making calls
/uat-execute .aiwg/testing/uat/plan.md --dry-run

# Skip issue filing
/uat-execute .aiwg/testing/uat/plan.md --no-issues

# Resume from a specific phase (after fixing a blocker)
/uat-execute .aiwg/testing/uat/plan.md --resume-from 3

# Override execution mode (run only quick smoke subset)
/uat-execute .aiwg/testing/uat/plan.md --execution-mode quick
```

## Parameters

### plan_path (required)
Path to the UAT plan document generated by `/uat-generate`.

### --mode (default: mcp)
Test mode. Must match the plan's mode. Currently only `mcp` is supported.

### --execution-mode (optional)
Override the plan's execution mode. Useful for running a quick smoke test from a full plan.

### --dry-run (optional)
Parse the plan and show what would be executed without making any MCP calls.

### --no-issues (optional)
Disable automatic issue filing for failures. Results are still recorded.

### --resume-from (optional)
Resume execution from a specific phase number. Useful after fixing a blocker found mid-execution. Previous phase results are loaded from the checkpoint file.

## Workflow

### Step 1: Parse Plan

Read the UAT plan and extract:
- Phase list with prerequisites
- Test cases per phase with MCP call specs
- Variable wiring requirements
- Negative test isolation flags

```
Parsing UAT Plan...
  Plan: plan-gitea-20260227.md
  Phases: 12
  Test cases: 165
  Negative tests: 32 (will be isolated)
  Variables: 18 cross-phase references
```

### Step 2: Preflight Check

Before executing tests, verify:
- MCP server(s) are reachable
- Required tools are available
- Authentication is valid

```
Preflight Check...
  Server: gitea — Connected
  Tools available: 78/78
  Auth: Valid (user: roctibot)
  Ready to execute.
```

### Step 3: Execute Phases

For each phase in order:

```
═══════════════════════════════════════
Phase 3: Issue Tracking (12 tests)
═══════════════════════════════════════

  P03-001: Create Issue (Happy Path)
    Tool: mcp__gitea__create_issue
    Parameters: {owner: "test-org", repo: "test-repo", title: "UAT Test Issue", body: "..."}
    Response: {number: 1, title: "UAT Test Issue", state: "open"}
    ✅ Criterion 1: Returns issue with number field — PASS
    ✅ Criterion 2: title matches — PASS
    ✅ Criterion 3: state is "open" — PASS
    Stored: ISSUE_INDEX = 1
    Result: PASS (3/3)

  P03-002: Get Issue by Index
    Tool: mcp__gitea__get_issue_by_index
    Parameters: {owner: "test-org", repo: "test-repo", index: 1}
    Response: {number: 1, title: "UAT Test Issue", ...}
    ✅ Criterion 1: Returns issue matching ISSUE_INDEX — PASS
    ✅ Criterion 2: title is "UAT Test Issue" — PASS
    Result: PASS (2/2)

  P03-003: Create Issue — Missing Title (Negative)
    [Isolation: single call]
    Tool: mcp__gitea__create_issue
    Parameters: {owner: "test-org", repo: "test-repo", body: "No title"}
    Response: {error: "validation failed"}
    ✅ Criterion 1: Returns error — PASS
    ✅ Criterion 2: Error indicates missing title — PASS
    Result: PASS (2/2)

Phase 3 Summary: 12/12 passed (0 failed, 0 skipped)
Duration: 45 seconds
```

### Step 4: Handle Failures

On test failure:

1. Record the failure with full details
2. File an issue (unless `--no-issues`)
3. Continue to next test (don't stop)
4. If a stored variable was expected but not captured, mark dependent tests as `error`

```
  P04-007: Update Repository Description
    Tool: mcp__gitea__update_repo
    Parameters: {owner: "test-org", repo: "test-repo", description: "Updated"}
    Response: {error: "not found", message: "repository not found"}
    ❌ Criterion 1: Returns updated repository — FAIL
    Result: FAIL (0/1)
    Issue filed: #415 "UAT: P04-007 — Update Repository returns not found"
```

### Step 5: Cleanup Phase

The cleanup phase ALWAYS runs, even if earlier phases failed:

```
═══════════════════════════════════════
Phase 11: Cleanup (always runs)
═══════════════════════════════════════

  P11-001: Delete Test Repository
    Tool: mcp__gitea__delete_repo  (if available)
    ...
  P11-002: Delete Test Organization
    ...

Cleanup complete. 3/3 test entities removed.
```

### Step 6: Save Results

Write results to `.aiwg/testing/uat/results/` following the `uat-result.yaml` schema:

```
Results saved:
  File: .aiwg/testing/uat/results/results-gitea-20260227-143022.yaml
  Checkpoint: .aiwg/testing/uat/results/checkpoint-gitea-20260227.yaml

Generate report with:
  /uat-report .aiwg/testing/uat/results/results-gitea-20260227-143022.yaml
```

### Step 7: Summary

```
═══════════════════════════════════════════
UAT Execution Complete
═══════════════════════════════════════════

Plan: plan-gitea-20260227.md
Duration: 8 minutes 32 seconds

Results:
  Total tests: 165
  Passed: 159 (96.4%)
  Failed: 4 (2.4%)
  Skipped: 1 (0.6%)
  Errors: 1 (0.6%)

Issues filed: 4
  #415: P04-007 — Update Repository returns not found
  #416: P06-003 — Wiki page creation timeout
  #417: P08-002 — Release tag not created
  #418: P09-011 — Search returns stale results

Coverage: 76/78 tools tested (97.4%)
  Untested: mcp__gitea__cancel_repo_action_run (no active runs)
            mcp__gitea__stop_stopwatch (no active stopwatch)

═══════════════════════════════════════════
```

## Error Handling

### Plan Not Found

```
Error: UAT plan not found at {path}

Generate a plan first:
  /uat-generate --mode mcp
```

### MCP Server Unavailable

```
Error: MCP server {name} is not reachable.

The UAT plan requires this server. Check your MCP configuration.
Retry with: /uat-execute {plan_path}
```

### Checkpoint Recovery

If execution crashes mid-run, results are checkpointed per-phase:

```
Checkpoint detected: .aiwg/testing/uat/results/checkpoint-gitea-20260227.yaml
  Phases completed: 0-5
  Phases remaining: 6-11

Resume with: /uat-execute {plan_path} --resume-from 6
```

## References

- Agent: @$AIWG_ROOT/agentic/code/addons/uat-mcp/agents/uat-executor.md
- Schema: @$AIWG_ROOT/agentic/code/addons/uat-mcp/schemas/uat-result.yaml
- Template: @$AIWG_ROOT/agentic/code/addons/uat-mcp/templates/uat-executor-guide.md