ai-agent-approval-audit-completeness

# AI Agent Approval-Audit Completeness
Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

<!-- dual-compat-start -->
## Use When

- An auditor (SOC 2 Type II / ISO 27001 / HIPAA) asks: *"Show me every irreversible action your agents took in Q2; prove each one had a documented approval before execution."*
- Building the **completeness check** (also called the gap-detection job) that, for every irreversible action in the audit window, verifies an approval row exists, was signed by an authorised approver, occurred before the action, and is linked back to the action audit row.
- Producing the **SOC 2 Processing Integrity (PI1.1)** evidence pack — system processing is complete, accurate, timely, and authorised.
- Closing the loop between the approval mechanism (`ai-agent-action-approval-and-hitl`) and the tamper-evident audit log (`ai-agent-audit-log-integrity`) so that a missing or backfilled approval is detected automatically.

## Do Not Use When

- Designing the approval **mechanism / state machine** — that is `ai-agent-action-approval-and-hitl`.
- Designing the **tamper-evident storage** of the audit log — `ai-agent-audit-log-integrity`.
- Building **per-control SOC 2 collectors** generally — `ai-agent-soc2-controls`; this skill is the dedicated PI1.1 collector.
- Writing **policy narrative** (control description, RACI) — SRS engine.

## Required Inputs

- Action audit log with `event_class`, `reversibility ∈ {reversible, soft_reversible, irreversible}`, `tool_name`, `task_id`, `step_index`, `tenant_id`, `occurred_at`, `payload_ref`, hash-chained per `ai-agent-audit-log-integrity`.
- Approval table with `approval_id`, `request_id`, `approver_id`, `approver_role`, `approved_at`, `signature`, `linked_action_id`, `policy_version`.
- Tool registry with `tool_id → required_approval_level` (none / single-approver / dual-approver / break-glass).
- Approver allow-list per tenant (role → identity).
- Clock authority (NTP-disciplined, monotonic check) so `approved_at < action_occurred_at` is provable.

## Workflow

1. Read this `SKILL.md`.
2. Run the **completeness check** (§1) over the rolling window — joins the action audit log to the approval table, classifies every row.
3. Triage **gaps** (§2) — there are five canonical gap classes; each has a remediation path.
4. Produce the **completeness evidence pack** (§3) — signed pack with totals, gap report, sample of approvals (auditor-pulled), integrity-chain witness.
5. Auto-open **exceptions** for any gap that is not a benign legacy / migration artefact (§4).
6. Wire the job into the **evidence pipeline** (`ai-agent-evidence-automation`) on a monthly cadence with per-quarter rollup.
7. Apply anti-patterns (§5).

## Quality Standards

- **Zero unexplained gaps.** Every irreversible-action row in the window either has a matching approval row OR a documented exception that has been triaged.
- **Temporal proof.** Every approval timestamp is strictly before the linked action timestamp; clock-skew tolerance is documented (≤ 2s) and any breach is itself an exception.
- **Cryptographic linkage.** Approval row carries `linked_action_chain_pos` (the chain position of the audit row it authorised). Verification job re-derives the chain hash to confirm linkage.
- **Approver authority.** Every approver identity is verified against the approver allow-list **as of the policy_version** in force at approval time — not the current allow-list.
- **No self-approval.** Approver identity ≠ initiator identity. Dual-approval requires two distinct approvers.
- **Job heartbeat is itself evidence.** A missed run is an exception.

## Anti-Patterns

- Joining on `approval.linked_action_id` only; missing the chain-position cross-check. A backfilled approval would pass.
- Comparing `approved_at < action_occurred_at` with database clock instead of an authoritative monotonic clock. Backdating is undetected.
- Allow-list checked against the **current** state at job time. Approvers who have since been removed look as if they were never authorised.
- Treating "approval missing" the same as "approval explicit-bypass with kill-switch flip". Different controls; different evidence rows.
- One giant gap report at audit time. Auditor wants monthly cadence inside the window.

## Outputs

- Gap-detection job (Python) running monthly.
- Completeness report schema (totals, by-tool, by-tenant, gap rows).
- Completeness evidence pack format (PI1.1).
- Exception register entries auto-opened for unresolved gaps.
- Auditor portal endpoint returning the latest signed pack per window.

## Evidence Produced

| Category | Artifact | Format | Example |
|----------|----------|--------|---------|
| Compliance | Approval completeness report | JSON + signed PDF | `evidence/soc2/PI1.1/2026-Q2/completeness-report.json` |
| Compliance | Gap rows (per irreversible action without approval) | JSONL | `.../gaps.jsonl` |
| Compliance | Sample of approvals (auditor sample) | JSONL | `.../approvals-sample.jsonl` |
| Compliance | Integrity-chain witness for the window | JSON | `.../chain-witness.json` |
| Compliance | Quarterly rollup | tar.gz pack | `.../PI1.1-2026Q2.tar.gz` |
| Compliance | Exception entries (open / closed during window) | JSONL | `.../exceptions.jsonl` |

## References

- `references/gap-detection-job.md` — Full Python implementation of the completeness / gap-detection job.
- `references/completeness-evidence.md` — Evidence pack schema for SOC 2 PI1.1.
- Companions: `ai-agent-action-approval-and-hitl` (approval mechanism), `ai-agent-audit-log-integrity` (storage), `ai-agent-soc2-controls` (PI1.1 control narrative consumer), `ai-agent-evidence-automation` (pack pipeline), `ai-agent-control-testing-and-attestation` (test wrapper).

<!-- dual-compat-end -->

## §1 The Completeness Check

The check is a **set-difference + temporal-validity** join across two tables, executed inside the database for atomicity.

```sql
-- compliance/queries/approval_completeness.sql
WITH irreversible_actions AS (
  SELECT
    al.id            AS action_id,
    al.chain_pos     AS action_chain_pos,
    al.tenant_id,
    al.task_id,
    al.step_index,
    al.tool_name,
    al.actor_id      AS initiator_id,
    al.occurred_at   AS action_at,
    al.policy_version
  FROM action_audit_log al
  WHERE al.reversibility = 'irreversible'
    AND al.occurred_at >= :window_start
    AND al.occurred_at <  :window_end
),
matched AS (
  SELECT
    ia.*,
    ap.id            AS approval_id,
    ap.approver_id,
    ap.approver_role,
    ap.approved_at,
    ap.signature,
    ap.linked_action_chain_pos,
    ap.policy_version AS approval_policy_version
  FROM irreversible_actions ia
  LEFT JOIN approvals ap
    ON ap.linked_action_id = ia.action_id
)
SELECT
  *,
  CASE
    WHEN approval_id IS NULL                                   THEN 'gap_missing'
    WHEN approved_at >= action_at                              THEN 'gap_after_action'
    WHEN linked_action_chain_pos <> action_chain_pos           THEN 'gap_chain_mismatch'
    WHEN approver_id = initiator_id                            THEN 'gap_self_approval'
    WHEN approval_policy_version <> policy_version             THEN 'gap_policy_version'
    ELSE 'ok'
  END AS classification
FROM matched;
```

Approver authority is checked in a second pass in Python against the **historical** allow-list (versioned per `policy_version`), not the current state.

```python
# compliance/jobs/approval_completeness.py
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Iterable
import hashlib, json

from audit.chain import verify_slice
from policy.versions import resolve_allowlist_at

@dataclass(frozen=True)
class Gap:
    action_id: int
    classification: str
    detail: dict

class ApprovalCompletenessJob:
    def __init__(self, db, evidence_vault, clock_skew_seconds: int = 2):
        self.db = db
        self.vault = evidence_vault
        self.skew = clock_skew_seconds

    def run(self, window_start: datetime, window_end: datetime) -> dict:
        rows = self.db.execute(
            open("compliance/queries/approval_completeness.sql").read(),
            window_start=window_start, window_end=window_end,
        ).fetchall()

        totals = {"irreversible_actions": len(rows), "matched": 0, "gaps": 0}
        gap_buckets: dict[str, list[Gap]] = {}

        for r in rows:
            cls = r["classification"]
            # Second-pass: historical allow-list check
            if cls == "ok":
                allow = resolve_allowlist_at(r["tenant_id"], r["policy_version"])
                if r["approver_id"] not in allow.get(r["approver_role"], set()):
                    cls = "gap_unauthorised_approver"
            # Clock-skew check
            if cls == "gap_after_action":
                drift = (r["approved_at"] - r["action_at"]).total_seconds()
                if drift <= self.skew:
                    cls = "ok_within_skew_tolerance"

            if cls.startswith("gap_"):
                totals["gaps"] += 1
                gap_buckets.setdefault(cls, []).append(
                    Gap(action_id=r["action_id"], classification=cls, detail=dict(r))
                )
            else:
                totals["matched"] += 1

        # Integrity witness over the audit-log slice
        witness = verify_slice(window_start, window_end)

        report = {
            "control_id": "PI1.1",
            "window_start": window_start.isoformat(),
            "window_end":   window_end.isoformat(),
            "totals":       totals,
            "gap_classes":  {k: len(v) for k, v in gap_buckets.items()},
            "chain_witness": witness,
            "policy_versions_observed": sorted({r["policy_version"] for r in rows}),
            "produced_at":  datetime.utcnow().isoformat(),
        }

        # Persist the report + gap rows + a sampled approvals JSONL
        report_path = self.vault.put_json("PI1.1", window_start, window_end, "completeness-report.json", report)
        gap_path    = self.vault.put_jsonl("PI1.1", window_start, window_end, "gaps.jsonl",
                                           [g.detail for bucket in gap_buckets.values() for g in bucket])
        sample_path = self.vault.put_jsonl("PI1.1", window_start, window_end, "approvals-sample.jsonl",
                                           self._sample_approvals(rows, n=200))

        # Open exceptions for each non-benign gap
        for cls, gaps in gap_buckets.items():
            for g in gaps:
                self._open_exception(g, cls)

        return {
            "report_path": report_path, "gap_path": gap_path, "sample_path": sample_path,
            "totals": totals, "gap_classes": report["gap_classes"],
        }

    def _sample_approvals(self, rows, n: int) -> list[dict]:
        import random
        ok = [r for r in rows if r["classification"] == "ok"]
        return random.sample(ok, min(n, len(ok)))

    def _open_exception(self, gap: Gap, cls: str):
        from compliance.exceptions import open_exception
        open_exception(
            control_id="PI1.1",
            severity={"gap_missing": "high", "gap_after_action": "high",
                      "gap_chain_mismatch": "critical", "gap_self_approval": "critical",
                      "gap_unauthorised_approver": "critical",
                      "gap_policy_version": "medium"}.get(cls, "medium"),
            title=f"Approval completeness gap: {cls} on action {gap.action_id}",
            evidence_ref=f"PI1.1/{gap.action_id}",
            detail=gap.detail,
        )
```

Full implementation including the SQL, second-pass policy resolver, exception opener, and the evidence-pack writer in `references/gap-detection-job.md`.

## §2 Gap Classes and Remediation

| Class | Meaning | Remediation |
|---|---|---|
| `gap_missing` | Irreversible action has no approval row | Investigate: bypass via kill-switch? backfill? Bug? Open exception; if explicit kill-switch flip, link kill-switch record and reclassify. |
| `gap_after_action` | Approval `approved_at >= action_at` (beyond skew tolerance) | Backdated or out-of-order. Critical for PI1.1. |
| `gap_chain_mismatch` | Approval's `linked_action_chain_pos` ≠ action's actual chain position | Approval references a different action; usually a row-ID reuse bug. Critical. |
| `gap_self_approval` | Approver = initiator | Segregation-of-duties failure. Critical. |
| `gap_unauthorised_approver` | Approver not on the allow-list as of `policy_version` | Identity / role drift. Critical. |
| `gap_policy_version` | Approval signed under a different policy version than the action | Stale approval; usually means policy changed mid-flight. Medium. |

## §3 Completeness Evidence Pack

The pack is a tar.gz containing:

```
PI1.1-2026Q2/
├── manifest.json                   # control_id, window, signer, sha256 of every file
├── completeness-report.json        # totals + gap class counts + chain witness
├── gaps.jsonl                      # one row per gap with full context
├── approvals-sample.jsonl          # sampled ok approvals (n=200) for auditor inspection
├── chain-witness.json              # audit-log chain seal at window_start / window_end
├── exceptions.jsonl                # exceptions opened during window + closure refs
├── attestation.txt                 # control-owner signature over manifest.sha256
└── signature.sig                   # ed25519 signature of manifest.json
```

The auditor pulls the pack at one URL:

```
GET /audit/soc2/control/PI1.1?window_start=2026-04-01&window_end=2026-06-30
```

Full schema in `references/completeness-evidence.md`.

## §4 Exception Lifecycle

A gap row opens a `compliance_exceptions` entry (schema in `ai-agent-soc2-controls`). The control owner triages within SLO (critical: 5 business days, high: 10, medium: 20). Closure requires a remediation note + evidence ref. Open exceptions appear in the quarterly attestation pack; the auditor will scope a finding if they exceed SLO.

## §5 Anti-Patterns

- Running the completeness job only at audit time. Detection is too late; gaps cannot be remediated retroactively.
- Defining "approval" purely by row existence. Cryptographic linkage to the chain position is what makes the evidence resistant to backfill.
- Allow-listing approvers from "the current role table." Use the policy version that was active at approval time.
- Treating every `gap_missing` as a critical incident. Some are explicit kill-switch flips with a documented bypass; reclassify those as `bypass_documented` and link to the kill-switch record.
- Sampling 5 approvals. Auditors typically pull 25 minimum per quarter; default to n=200 and let them sample.
- Skipping the chain witness. Without it, an auditor cannot rule out wholesale audit-log rewrite.