ds-verify

Announce: "Using ds-verify (Phase 5) to confirm reproducibility and completion."

## Contents

- [The Iron Law of DS Verification](#the-iron-law-of-ds-verification)
- [Verification Facts](#verification-facts)
- [The Verification Gate](#the-verification-gate)
- [Verification Checklist](#verification-checklist)
- [Reproducibility Demonstration](#reproducibility-demonstration)
- [Claims Requiring Evidence](#claims-requiring-evidence)
- [Insufficient Evidence](#insufficient-evidence)
- [Required Output Structure](#required-output-structure)
- [Completion Criteria](#completion-criteria)

## Context Monitoring

| Level | Remaining Context | Action |
|-------|------------------|--------|
| Normal | >35% | Proceed normally |
| Warning | 25-35% | Complete current review cycle, then trigger ds-handoff |
| Critical | ≤25% | Immediately trigger ds-handoff — do not start new review cycles |

# Verification Gate

Final verification with reproducibility checks and user acceptance interview.

<EXTREMELY-IMPORTANT>
## The Iron Law of DS Verification

**NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION. This is not negotiable.**

**Load shared enforcement first.**

Auto-load all constraints matching `applies-to: ds-verify`:

!`uv run python3 ${CLAUDE_SKILL_DIR}/../../scripts/load-constraints.py ds-verify`

**You MUST have these constraints loaded before proceeding. No claiming you "remember" them.**

Before claiming analysis is complete, you MUST:
1. RE-RUN - Execute analysis fresh (not cached results)
2. CHECK - Verify outputs match expectations
3. REPRODUCE - Confirm results are reproducible
4. ASK - Interview user about constraints and acceptance
5. Only THEN claim completion

This applies even when:
- "I just ran it"
- "Results look the same"
- "It should reproduce"
- "User seemed happy earlier"

**If you catch yourself thinking "I can skip verification," STOP — you're about to deliver unverified results that waste the user's time.**
</EXTREMELY-IMPORTANT>

## Verification Facts

- Review checks methodology; verify checks reproducibility. They are different gates — a passed review is not evidence the analysis reproduces.
- Reproducibility means re-running, not re-reading: prior results don't prove current reproducibility, because code, data, or environment may have changed since. Reading cached output proves nothing.
- Verifying your own work shares the implementer's biases — the reproducibility demonstration is run fresh, not vouched for by its author. "Verified" without re-execution is an unverified claim.
- The 10-minute reproducibility check is cheap against the 10 days of debugging when someone else can't run the analysis — skipping it to deliver faster is anti-efficient.
- "User will be happy" is an assumption, not acceptance — completion requires the user's explicit confirmation, not your prediction of it.

## Static Analysis (Constraint Check Scripts)

Before running runtime DQ checks, run the static analysis constraint check suite:

```bash
bash "${CLAUDE_SKILL_DIR}/../../scripts/check-all-ds.sh" "$(pwd)"
```

This runs all DS constraint check scripts (determinism, join audits, idempotency, error handling, schema contracts, standard errors, visualization integrity).

**If any check FAILS:** Report the failures in LEARNINGS.md. These are code quality issues in the analysis scripts that must be fixed before proceeding. Dispatch a fix subagent if needed.

**If all checks PASS:** Proceed to runtime DQ checks.

## The Verification Gate

**Checkpoint type:** decision (user confirms results — cannot auto-advance)

Before making ANY completion claim, follow this flowchart.

**This flowchart IS the specification. If prose elsewhere and this diagram disagree, the diagram wins.**

```
        ┌──────────────────────────────┐
        │ 1. RE-RUN (fresh, not cached) │
        └──────────────┬───────────────┘
                       ▼
        ┌──────────────────────────────┐
        │ 2. CHECK vs success criteria  │
        └──────────────┬───────────────┘
                  pass? │
            ┌───── no ──┴── yes ─────┐
            ▼                        ▼
   ┌─────────────────┐   ┌──────────────────────────┐
   │ NEEDS WORK →    │   │ 3. REPRODUCE             │
   │ log + dispatch  │   │ (same inputs→same outputs)│
   │ fix subagent    │   └────────────┬─────────────┘
   └────────┬────────┘        match?  │
            │           ┌──── no ──────┴── yes ───┐
            │           ▼                         ▼
            │  ┌─────────────────┐   ┌─────────────────────────┐
            │  │ NEEDS WORK →    │   │ 4. ASK — user           │
            │  │ non-determinism │   │ acceptance interview    │
            │  │ is a defect     │   └───────────┬─────────────┘
            │  └────────┬────────┘    accept?     │
            │           │      ┌── no/partial ────┴── yes ──┐
            │           │      ▼                            ▼
            │           │  ┌──────────────────┐  ┌────────────────────┐
            └───────────┴─▶│ loop: ds-fix /   │  │ 5. CLAIM COMPLETE  │
                           │ ds-implement,    │  │ (only after 1-4)   │
                           │ then re-verify   │  └────────────────────┘
                           └──────────────────┘
```

**Skipping any step is not verification.** Reaching step 5 without passing 1-4 is a false completion claim.

## Visual Diagnostics for Verification

When presenting verification results to the user in the acceptance interview, generate diagnostic plots to support the decision:

| Verification Check | Diagnostic to Generate |
|-------------------|----------------------|
| Reproducibility comparison | Overlay plot of Run 1 vs Run 2 key outputs |
| Data integrity | Pipeline waterfall chart (input rows → cleaning → joins → final) |
| Distribution sanity | Histogram/density plots of key variables with expected ranges annotated |
| Model performance | ROC curve, residual plot, or coefficient comparison (as appropriate) |

**Format:** Inline plots in notebooks, or saved to `scratch/diagnostics/` for script-based workflows. Present alongside the acceptance interview questions.

## Verification Checklist

### Technical Verification

#### Outputs Match Expectations
- [ ] All required outputs generated
- [ ] Output formats correct (files, figures, tables)
- [ ] Numbers are reasonable (sanity checks)
- [ ] Visualizations render correctly

#### Reproducibility Confirmed
- [ ] Ran analysis twice, got same results
- [ ] Random seeds produce consistent output
- [ ] No dependency on execution order
- [ ] Environment documented (packages, versions)

#### Data Integrity
- [ ] Input data unchanged
- [ ] Row counts traceable through pipeline
- [ ] No silent data loss or corruption

**Trace to Requirements:** For each success criterion, reference its requirement ID (e.g., "DATA-01: Panel has 50K+ firm-years — VERIFIED with df.shape output"). End-to-end traceability from SPEC.md through PLAN.md through VALIDATION.md through verification.

### User Acceptance Interview

**CRITICAL:** Before claiming completion, conduct user interview.

#### Step 1: Replication Constraints

```
AskUserQuestion:
  question: "Were there specific methodology requirements I should have followed?"
  options:
    - label: "Yes, replicating existing analysis"
      description: "Results should match a reference"
    - label: "Yes, required methodology"
      description: "Specific methods were mandated"
    - label: "No constraints"
      description: "Methodology was flexible"
```

If replicating:
- Ask for reference to compare against
- Verify results match within tolerance
- Document any deviations and reasons

#### Step 2: Results Verification

```
AskUserQuestion:
  question: "Do these results answer your original question?"
  options:
    - label: "Yes, fully"
      description: "Analysis addresses the core question"
    - label: "Partially"
      description: "Some aspects addressed, others missing"
    - label: "No"
      description: "Does not answer the question"
```

If "Partially" or "No":
1. Ask which aspects are missing
2. Return to `/ds-implement` to address gaps
3. Re-run verification

#### Step 3: Output Format

```
AskUserQuestion:
  question: "Are the outputs in the format you need?"
  options:
    - label: "Yes"
      description: "Format is correct"
    - label: "Need adjustments"
      description: "Format needs modification"
```

#### Step 4: Confidence in Results

```
AskUserQuestion:
  question: "Do you have any concerns about the methodology or results?"
  options:
    - label: "No concerns"
      description: "Comfortable with approach and results"
    - label: "Minor concerns"
      description: "Would like clarification on some points"
    - label: "Major concerns"
      description: "Significant issues need addressing"
```

## Reproducibility Demonstration

**MANDATORY:** Demonstrate reproducibility before completion.

<EXTREMELY-IMPORTANT>
## Independent Verification Required

**You MUST NOT verify your own work. Spawn a fresh Task agent for reproducibility.**

The implementer shares biases and sunk-cost attachment. A fresh subagent sees only the spec and outputs — it verifies without context pollution.

**If you're about to re-run the analysis yourself, STOP. Dispatch a Task agent.**
</EXTREMELY-IMPORTANT>

Dispatch a fresh Task agent to run the reproducibility check:

*All paths below are relative to this skill's base directory.*
```
Agent(subagent_type="general-purpose",
  allowed_tools=["Read", "Glob", "Grep", "Bash(read-only)"],
  prompt="""
# Reproducibility Verification

**Tool Restrictions:** The verifier is READ-ONLY. It re-runs analyses and checks output but MUST NOT modify notebooks, scripts, or code. It MUST NOT use Write or Edit.

Verify this analysis produces consistent results from a fresh run.

## Context
- Read .planning/SPEC.md for objectives and success criteria
- Read .planning/PLAN.md for expected outputs
- Read .planning/LEARNINGS.md for pipeline documentation

## Shared Checks
Read the shared check definitions:
Read `${CLAUDE_SKILL_DIR}/../../skills/ds-implement/references/ds-checks.md` and follow its instructions.

Run checks: DQ1-DQ4, DQ6, M1, R1

## Reproducibility Protocol

### For scripts:
```python
# Run 1
result1 = run_analysis(seed=42)
hash1 = hash(str(result1))

# Run 2
result2 = run_analysis(seed=42)
hash2 = hash(str(result2))

# Verify
assert hash1 == hash2, "Results not reproducible!"
print(f"Reproducibility confirmed: {hash1} == {hash2}")
```

### For notebooks:
```bash
jupyter nbconvert --execute --inplace notebook.ipynb
papermill notebook.ipynb output.ipynb -p seed 42
```

## Required Checks
1. RE-RUN: Execute analysis fresh (not cached results)
2. CHECK: Verify outputs match SPEC.md success criteria
3. REPRODUCE: Same inputs → same outputs (run twice, compare hashes)
4. DATA INTEGRITY: Input data unchanged, row counts traceable

## Output
Report:
- Reproducibility: PASS/FAIL (with hash comparison)
- Data quality checks: DQ1-DQ4, DQ6 results
- Spec compliance: M1 result
- Any discrepancies found
""")
```

**Post-subagent boundary (C5):** After verification agent returns, read its report only. Do NOT read source code, notebooks, or data files yourself. If FAIL, dispatch a fresh investigation subagent.

**If Task agent reports FAIL:** Dispatch a fresh Task agent to investigate the discrepancy. Do NOT investigate yourself — that violates the post-subagent boundary (C5 from ds-common-constraints.md).

## Claims Requiring Evidence

| Claim | Required Evidence |
|-------|-------------------|
| "Analysis complete" | All success criteria verified |
| "Results reproducible" | Same output from fresh run |
| "Matches reference" | Comparison showing match |
| "Data quality handled" | Documented cleaning steps |
| "Methodology appropriate" | Assumptions checked |

## Insufficient Evidence

These do NOT count as verification:

- Previous run results (must be fresh)
- "Should be reproducible" (demonstrate it)
- Visual inspection only (quantify where possible)
- Single run (need reproducibility check)
- Skipped user acceptance (must ask)

## Required Output Structure

```markdown
## Verification Report: [Analysis Name]

### Technical Verification

#### Outputs Generated
- [ ] Output 1: [location] - verified [date/time]
- [ ] Output 2: [location] - verified [date/time]

#### Reproducibility Check
- Run 1 hash: [value]
- Run 2 hash: [value]
- Match: YES/NO

#### Environment
- Python: [version]
- Key packages: [list with versions]
- Random seed: [value]

### User Acceptance

#### Replication Check
- Constraint: [none/replicating/required methodology]
- Reference: [if applicable]
- Match status: [if applicable]

#### User Responses
- Results address question: [yes/partial/no]
- Output format acceptable: [yes/needs adjustment]
- Methodology concerns: [none/minor/major]

### Verdict

**COMPLETE** or **NEEDS WORK**

[If COMPLETE]
- All technical checks passed
- User accepted results
- Reproducibility demonstrated

[If NEEDS WORK]
- [List items requiring attention]
- Recommended next steps
```

## Workflow Loops (If NEEDS WORK)

1. Identify which item(s) need fixing
2. Return to ds-implement with specific task(s) to fix
3. Re-run those tasks with output-first verification
4. Update LEARNINGS.md with fixes
5. Re-invoke ds-verify for fresh verification

**Maximum 3 verification cycles.** If issues persist after 3 rounds, escalate to user with summary of blocking issues.

**Chaining instruction (if NEEDS WORK).** Discover and load ds-implement:
Read `${CLAUDE_SKILL_DIR}/../../skills/ds-implement/SKILL.md` and follow its instructions.
Then fix the identified issues and re-run verification.

## Completion Criteria

**Only claim COMPLETE when ALL are true:**

- [ ] All success criteria from SPEC.md verified
- [ ] Results reproducible (demonstrated, not assumed)
- [ ] User confirmed results address their question
- [ ] User has no major concerns
- [ ] Outputs in acceptable format
- [ ] If replicating: results match reference

**Both technical and user acceptance must pass. No shortcuts.**

## Workflow Complete

When user confirms all criteria are met:

**Announce:** "DS workflow complete. All 5 phases passed."

The `/ds` workflow is now finished. Offer to:
- Export results to final format
- Clean up `.planning/` files
- Start a new analysis with `/ds`