factcheck-guard

# FactCheck Guard — Runtime Claim Verification

Prevents false claims about the codebase. Grep results are hypotheses, not proof.
Every assertion about "what the code does" must be verified by reading the actual code.

## Why This Exists

An 80% false claim rate was observed when agents trusted grep results without reading files.
The pattern repeats: grep finds a string, agent asserts behavior, behavior is wrong.
This skill converts the claim-verification guidelines into an actionable protocol.

## Confidence Markers (MANDATORY)

Every factual claim about the codebase MUST carry one of these markers:

| Marker | Meaning | Requirements |
|--------|---------|-------------|
| [VERIFIED] | Read the file, traced the logic | Must cite file:line |
| [INFERRED] | Based on grep or file name | Must verify before asserting to others |
| [UNCERTAIN] | Not yet checked | Must investigate before making the claim |

Never promote an [INFERRED] claim to a statement of fact.
Never let [UNCERTAIN] appear in a response to the user — investigate first.

---

## Verification Protocol

### Level 1: Existence Claims

When asserting "X exists in the codebase":

**Step 1 — Search (produces [INFERRED])**
```
Grep/Glob for the pattern.
Result: "X appears at path/to/file.ts:42" [INFERRED]
```

**Step 2 — Read the file (required to reach [VERIFIED])**
```
Read path/to/file.ts, locate line 42.
Confirm the match is actual code, not a comment or string literal.
```

**Step 3 — Check context**
```
Is the code in a live code path, or inside a dead branch / disabled block?
Is it imported and actually used?
Does the function name match what you expected, or is it a different concept?
```

**Step 4 — State the claim with citation**
```
"Error handling exists at src/api/handler.ts:67 — try/catch wraps the database call" [VERIFIED]
```

---

### Level 2: Absence Claims

When asserting "X does NOT exist" — absence claims are the most dangerous.
A single missed search term invalidates the entire claim.

**Step 1 — Primary search**
```
Grep for the primary term. No results found.
Status: [INFERRED ABSENT] — not yet VERIFIED
```

**Step 2 — Alternative terms (minimum 3)**
```
Different naming conventions, abbreviations, synonyms.

Example: Searching for "rate limiting"
Try: "rateLimit", "rate_limit", "throttle", "throttling", "requestLimit", "limiter"
If any match: investigate that file before concluding
```

**Step 3 — Check likely locations manually**
```
Where would X logically live if it existed?
Read those files directly — grep may miss dynamically constructed patterns.

Example: Checking for auth middleware
Read: src/middleware/, app/middleware.ts, server.ts, app.ts
Even if grep found nothing, scan these files for auth-related logic.
```

**Step 4 — Only then claim VERIFIED ABSENT**
```
"Rate limiting does not exist. Searched: rateLimit, rate_limit, throttle, throttling,
requestLimit, limiter. Checked: src/middleware/index.ts, server.ts, app.ts. None found." [VERIFIED ABSENT]
```

---

### Level 3: Behavior Claims

When asserting "the system does X" — the most subtle failure mode.
Finding a function name does not tell you what the function actually does.

**Step 1 — Locate the function**
```
Grep/Glob finds the function name.
Status: [INFERRED] — you know it exists, not what it does
```

**Step 2 — Read the full function body**
```
Read the file. Read the entire function, not just the signature.
Look for: branches, early returns, side effects, what it actually returns.
```

**Step 3 — Trace the logic path**
```
Follow the execution path relevant to your claim.
If the function calls other functions, read those too (one level deep minimum).
Check: what happens in the error path? What are the edge cases?
```

**Step 4 — State behavior with evidence**
```
"processPayment() validates card number via Luhn check (payment.ts:34),
then calls Stripe API (payment.ts:41), and on failure rolls back the DB
transaction (payment.ts:58) — it does NOT charge the card on validation failure." [VERIFIED]
```

---

## Common False Claim Patterns

| Pattern | Why It Fails | Correct Approach |
|---------|-------------|-----------------|
| grep found "try.*catch" | Could be in comment, string, or test mock | Read the file, confirm it wraps real code |
| grep found nothing | Wrong term, wrong directory, different naming | Try 3+ alternative terms |
| File named auth.ts exists | Doesn't mean auth is implemented | Read the file |
| Type definition has a field | Doesn't mean the field is set or used | Grep for field assignment and usage |
| Function is called somewhere | Doesn't mean it does what the name implies | Read the function body |
| Found in __tests__ directory | Test mocks are not real implementations | Confirm the actual source file |
| grep -L returned files | Those files lack the pattern — but pattern might exist under a different name | Try alternative terms before concluding |
| Found in commented-out block | Not active code | Check if block is enabled |

---

## Two-Pass Audit Protocol

For system audits, security reviews, and architecture reviews:

### Pass 1: Generate Hypotheses

Go through each area systematically. Produce hypothesis statements only:

```
"Input validation might be missing from the /api/upload route" [INFERRED]
"The auth token might not be refreshed on expiry" [INFERRED]
"Database queries might not use parameterized statements" [INFERRED]
```

Do not state these as facts. Do not include them in the user-facing summary.

### Pass 2: Verify Each Hypothesis

For every [INFERRED] hypothesis:

```
Hypothesis: "Input validation missing from /api/upload" [INFERRED]

Action: Read src/app/api/upload/route.ts
Found: Zod schema validation at line 18, applied before file processing.
Result: "Input validation exists at api/upload/route.ts:18" [VERIFIED]
Hypothesis was wrong.
```

```
Hypothesis: "Database queries might not be parameterized" [INFERRED]

Action: Search for raw SQL patterns (string interpolation in queries)
Grep: `db.query(\`.*\${`, `db.execute(\`.*\${`
Read: src/lib/db.ts, src/repositories/*.ts
Found: All queries use prepared statements. One raw query at analytics.ts:89.
Result: "Raw SQL injection risk at src/analytics.ts:89" [VERIFIED]
```

Only after Pass 2 are claims safe to report.

---

## Quick Checklist

Before making any codebase claim, confirm:

- [ ] Have I read the actual file, not just grep results?
- [ ] Is the code in a live path (not commented out, not test-only)?
- [ ] For absence claims: did I try 3+ alternative search terms?
- [ ] For absence claims: did I manually check the likely locations?
- [ ] For behavior claims: did I read the full function, not just the name?
- [ ] Does my claim include a file:line citation?
- [ ] Is my marker accurate: [VERIFIED] / [INFERRED] / [UNCERTAIN]?

---

## Integration Points

**claim-verification.md** — the rule file provides guidelines; this skill provides the action protocol.

**Agents that must use this skill:**
- code-reviewer: before asserting "the code lacks X"
- security-reviewer: before asserting "vulnerability exists/doesn't exist"
- architect: before asserting "pattern is/isn't used"
- sleuth: before asserting "root cause is X"
- coroner: before asserting "same bug exists elsewhere"

**Hook integration (optional):**
After every Grep tool result, the `epistemic-reminder` hook injects a warning.
That warning is this skill's Level 1 trigger — search results are [INFERRED], not [VERIFIED].

---

## Examples: Wrong vs Right

**Existence claim — WRONG**
```
Grep found "authMiddleware" in 3 files.
Claim: "Auth middleware is applied to all routes."
```

**Existence claim — RIGHT**
```
Grep found "authMiddleware" in 3 files.
Read server.ts: middleware applied only to /api/* routes, not /public/*.
Read admin/router.ts: authMiddleware missing from /admin/health endpoint.
Claim: "Auth middleware applied to /api/* (server.ts:23) but missing from
/admin/health (admin/router.ts:8)." [VERIFIED]
```

**Absence claim — WRONG**
```
Grep "rate limit" returned no results.
Claim: "No rate limiting implemented."
```

**Absence claim — RIGHT**
```
Grep "rate limit" → no results.
Grep "rateLimit", "throttle", "limiter", "requestLimit" → no results.
Read: src/middleware/index.ts, server.ts, next.config.js.
Confirmed none contain request throttling logic.
Claim: "No rate limiting found after searching 6 term variants and
reading 3 likely files." [VERIFIED ABSENT]
```