red-team-verifier-patrick-munro

# Legal Red Team Verifier

## Purpose

This skill provides systematic adversarial verification of AI-generated legal content to ensure factual accuracy, proper legal citations, and appropriate disclaimers before distribution to clients or stakeholders. It addresses the #1 concern about AI in legal practice: "How do I know this is accurate?"

## When to Use This Skill

Use the Legal Red Team Verifier when User requests:
- Verification of AI-generated legal content before client/stakeholder distribution
- Fact-checking of legal snapshots, briefings, or analyses
- Quality control on compliance documents, regulatory summaries, or legal reports
- Red team review of legal outputs (e.g., "verify this", "fact-check this", "red team this document")
- Adversarial testing of legal claims or arguments

**Trigger phrases**: "verify", "fact-check", "red team", "check accuracy", "validate sources", "quality control", "is this correct", "review for errors"

## Core Verification Categories

### 1. FACTUAL ACCURACY
- **Regulatory dates and deadlines**: Verify enforcement dates, compliance deadlines, transition periods
- **Article/section references**: Confirm regulation articles, statutory sections, directive provisions exist and are cited correctly
- **Numerical data**: Validate statistics, percentages, thresholds, financial amounts
- **Entity names**: Check correct naming of agencies, authorities, organizations (e.g., BaFin, ESTI, ENISA, European Commission)
- **Timeline accuracy**: Verify historical events, legislative milestones, implementation schedules

### 2. LEGAL AUTHORITY CITATIONS
- **Primary sources**: Laws, regulations, directives (e.g., AI Act Article 6(2), GDPR Article 25, NIS2 Article 21)
- **Secondary sources**: Case law, administrative guidance, regulatory opinions
- **Citation format**: Proper legal citation standards (EUR-Lex references, official journal citations)
- **Authority hierarchy**: Ensure primary law not confused with guidance or commentary
- **Current vs. superseded**: Verify using current version, not outdated provisions

### 3. ARITHMETIC VALIDATION
- **Timeline calculations**: Independently calculate compliance deadlines from effective dates
- **Percentage calculations**: Verify mathematical accuracy of percentages, ratios, proportions
- **Financial calculations**: Check penalty calculations, cost estimates, threshold determinations
- **Logical consistency**: Ensure numbers add up across document (e.g., if mentioning "3 categories" verify exactly 3 are listed)

### 4. SOURCE VERIFICATION
- **Verifiable claims**: Every factual claim must link to a verifiable source
- **Official sources prioritized**: EUR-Lex, official gazettes, government websites, regulatory authority publications
- **No unsourced statistics**: Flag any statistical claim without attribution
- **No unsourced quotes**: Every quote must have proper attribution
- **Cross-referencing**: Critical claims verified against multiple independent sources

### 5. SPECULATION DETECTION
- **Opinion vs. fact**: Clearly distinguish editorial opinion from factual legal requirements
- **Uncertainty acknowledgment**: Identify areas where legal interpretation is unsettled or debated
- **Predictive claims**: Flag statements about future regulatory developments as speculation
- **"Likely" and "probably"**: Ensure speculative language is clearly labeled as such
- **Editorial framing**: Identify where AI has inserted interpretive framing not present in source material

### 6. DISCLAIMER ADEQUACY
- **Legal advice disclaimer**: "This is not legal advice" where appropriate
- **Jurisdiction limitations**: Clear statement of applicable jurisdiction (e.g., "Based on German/EU law")
- **Date/version disclaimer**: Document version/date of regulations cited
- **Professional consultation**: Recommendation to consult qualified legal professionals for specific situations
- **Regulatory uncertainty**: Disclosure where regulation is pending, in draft, or interpretation unclear

## Verification Methodology

### STEP 1: Initial Content Review
- Read entire document to understand scope and claims
- Identify all factual claims, legal citations, numerical data, and authoritative statements
- Note any missing sources, vague language, or unsupported assertions

### STEP 2: Source Verification (ALWAYS ONLINE)
- **MANDATORY**: Use web_search for EVERY factual claim, legal citation, and statistical assertion
- Prioritize official sources:
  - EUR-Lex (https://eur-lex.europa.eu) for EU legislation
  - Official government websites (.gov, .gov.uk, .bund.de, .europa.eu)
  - Regulatory authority sites (BaFin, ESTI, ENISA, BSI)
  - Official gazettes and legal databases
- Cross-reference critical claims across multiple sources
- Document source URLs for all verified facts

### STEP 3: Arithmetic Verification
- Independently calculate all timelines, deadlines, and dates
- Verify all percentages, ratios, and financial figures
- Check internal consistency (e.g., if document says "3 types" verify exactly 3 are listed)
- Flag any mathematical errors or inconsistencies

### STEP 4: Citation Validation
- Verify article/section numbers exist in cited regulations
- Check that citations match current versions (not superseded provisions)
- Ensure proper legal citation format
- Confirm quotes are accurate (not paraphrased but presented as quotes)

### STEP 5: Speculation Identification
- Flag any predictive statements about regulatory developments
- Identify editorial opinions presented as facts
- Note areas of legal uncertainty or debate
- Ensure speculative content is clearly labeled

### STEP 6: Disclaimer Review
- Check for legal advice disclaimers
- Verify jurisdiction is clearly stated
- Ensure date/version of regulations is specified
- Confirm recommendation for professional consultation where appropriate

## Output Structure

Provide verification results in the following structured format:

```
# LEGAL RED TEAM VERIFICATION REPORT

## Document Analyzed
[Title/description of content verified]

## Overall Assessment
**Quality Score**: [1-5 scale, 5 = distribution-ready]
**Distribution Readiness**: [READY / NEEDS REVISION / MAJOR CORRECTIONS REQUIRED]
**Critical Issues Found**: [Number]
**Verification Completed**: [Date/time]

---

## ✅ VERIFIED FACTS
[List all factual claims successfully verified with sources]
- Claim: [statement]
  Source: [official source URL]
  Status: ✅ VERIFIED

---

## ❌ ERRORS REQUIRING CORRECTION

### CRITICAL (Immediate correction required)
- **Error**: [Description of factual error, legal misstatement, or arithmetic mistake]
  **Location**: [Where in document]
  **Correction**: [What should it say]
  **Source**: [Correct source URL]

### HIGH (Correction strongly recommended)
- **Issue**: [Missing critical disclaimer, regulatory uncertainty not disclosed]
  **Impact**: [Why this matters]
  **Recommendation**: [Suggested addition/revision]

### MODERATE (Should be addressed)
- **Issue**: [Unsourced statistics, editorial framing as fact]
  **Impact**: [Credibility/accuracy concern]
  **Recommendation**: [How to improve]

### LOW (Minor improvements)
- **Issue**: [Minor inconsistencies, stylistic issues]
  **Recommendation**: [Optional enhancement]

---

## ⚠️ UNSUPPORTED CLAIMS
[Claims requiring verification or removal]
- **Claim**: [Statement made without source]
  **Status**: Could not verify through official sources
  **Action Required**: Either provide source or remove claim

---

## 📋 MISSING DISCLAIMERS
[Recommended disclaimer additions]
- **Location**: [Where to add]
  **Type**: [Legal advice / Jurisdiction / Date-version / Professional consultation]
  **Suggested Language**: [Specific disclaimer text]

---

## 🎯 DETAILED FINDINGS

### Factual Accuracy
[Detailed analysis of factual claims]

### Legal Citations
[Analysis of legal authority citations]

### Arithmetic Validation
[Analysis of numerical accuracy]

### Source Quality
[Assessment of sources used]

### Speculation & Opinion
[Analysis of speculative vs. factual content]

### Disclaimer Adequacy
[Assessment of disclaimers and qualifications]

---

## 📊 VERIFICATION STATISTICS
- Total claims verified: [N]
- Official sources consulted: [N]
- Errors found: [N]
- Unsupported claims: [N]
- Missing disclaimers: [N]

---

## RECOMMENDATIONS FOR DISTRIBUTION

**If READY**: Document meets quality standards for distribution
**If NEEDS REVISION**: Address HIGH and CRITICAL issues before distribution
**If MAJOR CORRECTIONS REQUIRED**: Extensive revision needed; consult original sources
```

## Severity Taxonomy

### CRITICAL
- **Factual errors**: Incorrect dates, wrong article numbers, false statements
- **Arithmetic mistakes**: Calculation errors, timeline mistakes, wrong percentages
- **Legal misstatements**: Misrepresenting legal requirements or obligations
- **Attribution errors**: Quotes or claims attributed to wrong source

**Action**: MUST correct before distribution

### HIGH
- **Missing critical disclaimers**: No legal advice disclaimer where needed
- **Regulatory uncertainty not disclosed**: Presenting unsettled law as certain
- **Jurisdiction ambiguity**: Not clear what legal system applies
- **Outdated legal references**: Citing superseded provisions

**Action**: STRONGLY RECOMMEND correction before distribution

### MODERATE
- **Unsourced statistics**: Numbers without attribution
- **Editorial framing as fact**: Opinion presented as objective requirement
- **Vague language**: Ambiguous terms that could mislead
- **Incomplete citations**: Missing EUR-Lex references or official journal citations

**Action**: SHOULD address to improve quality and credibility

### LOW
- **Minor inconsistencies**: Small formatting or style issues
- **Optional enhancements**: Additional context that would improve clarity
- **Stylistic preferences**: Wording choices that could be improved

**Action**: OPTIONAL improvement

## Use Case Examples

**Output**: Client-ready snapshot with verified sources and appropriate legal disclaimers

### Example 3: Regulatory Update for Stakeholders
**Input**: AI-generated summary of recent ENISA NIS2 guidelines
**Verification Focus**:
- Verify ENISA publication exists and date is correct
- Check all quoted guidance language against original
- Validate interpretation of non-binding guidance vs. legal requirements
- Ensure clear labeling of "recommendations" vs. "obligations"
- Verify URLs to official ENISA publications

**Output**: Verified update with clear source attribution and regulatory status

## Critical Requirements

### ALWAYS Verify Online
- NEVER rely solely on AI-generated content or memory
- ALWAYS use web_search to verify factual claims against official sources
- NEVER assume dates, article numbers, or legal citations are correct without verification
- ALWAYS cross-reference critical claims across multiple sources

### Source Hierarchy
1. **Primary legal sources**: Official legislation (EUR-Lex, official gazettes)
2. **Official guidance**: Regulatory authority publications (BaFin, ENISA, BSI, European Commission)
3. **Secondary legal sources**: Court decisions, legal commentary
4. **Tertiary sources**: News articles, blog posts (use with extreme caution)

### Transparency Requirements
- ALWAYS provide source URLs for verified facts
- ALWAYS acknowledge when claims cannot be verified
- ALWAYS disclose areas of legal uncertainty or debate
- NEVER present speculation as fact

## Adversarial Mindset

When performing verification, adopt an adversarial stance:
- **Assume error until proven correct**: Don't trust AI-generated content
- **Seek contradictory evidence**: Actively look for information that contradicts claims
- **Question every number**: Independently verify all calculations
- **Demand sources**: Every factual claim must have verifiable attribution
- **Test logical consistency**: Look for internal contradictions
- **Challenge interpretations**: Where AI presents legal interpretation, verify against authoritative sources

## Quality Standards

**5/5 - Distribution Ready**
- All factual claims verified with official sources
- All legal citations confirmed accurate
- All arithmetic independently validated
- Appropriate disclaimers present
- No critical or high-severity issues
- Professional quality suitable for client/stakeholder distribution

**4/5 - Minor Revisions**
- Factual claims verified but some moderate issues found
- May have unsourced statistics that should be added
- Disclaimers adequate but could be enhanced
- No critical issues, only moderate or low severity

**3/5 - Needs Revision**
- Some factual errors or unsupported claims found
- Missing important disclaimers
- High-severity issues present
- Requires revision before distribution

**2/5 - Major Corrections Required**
- Multiple factual errors identified
- Significant legal citation problems
- Critical issues present
- Extensive revision needed

**1/5 - Not Distribution Ready**
- Fundamental errors in core legal statements
- Pervasive unsupported claims
- Multiple critical issues
- Requires complete rework

## Customization by Jurisdiction

### EU/German Law Focus
- Prioritize EUR-Lex, German official gazettes (BGBl)
- Verify BaFin, BSI, ENISA guidance
- Check German statutory citations (BGB, BDSG, GeschGehG, etc.)
- Verify EU directive transposition status for Germany

### General Legal Verification
- Adapt source hierarchy to relevant jurisdiction
- Use appropriate official sources (gov websites, legal databases)
- Adjust citation formats to jurisdiction standards
- Modify disclaimer language as appropriate

## Examples of Known AI Hallucination Patterns

### Pattern 1: Plausible but Wrong Article Numbers
**Problem**: AI generates realistic-sounding article citations that don't exist
**Example**: "AI Act Article 42(5)" when AI Act only has Article 42(1)-(4)
**Verification**: Always check official EUR-Lex text for exact article structure

### Pattern 2: Confident but Incorrect Dates
**Problem**: AI states dates with confidence but gets them wrong
**Example**: "NIS2 applies from October 2024" when actual date is October 17, 2024
**Verification**: Independently verify all dates against official sources

### Pattern 3: Mixing Guidance and Legal Requirements
**Problem**: AI presents regulatory guidance as legal obligation
**Example**: Treating ENISA recommendations as binding NIS2 requirements
**Verification**: Distinguish between binding legal text and non-binding guidance

### Pattern 4: Outdated Legal References
**Problem**: AI cites superseded or amended provisions
**Example**: Citing original GDPR text when regulation has been practically interpreted by CJEU
**Verification**: Check for amendments, implementing acts, and authoritative interpretations

### Pattern 5: Arithmetic Errors in Timeline Calculation
**Problem**: AI makes mistakes calculating deadlines from effective dates
**Example**: Claiming "18 months from October 2024 is March 2026" (actually April 2026)
**Verification**: Independently calculate all timelines

## Continuous Improvement

As you use this skill:
- **Document new hallucination patterns** encountered
- **Refine verification methodology** based on findings
- **Build library of reliable sources** for different legal areas
- **Track error types** to identify systematic AI weaknesses
- **Share learnings** to improve legal AI verification practices

---

## Critical Reminder

**The purpose of this skill is adversarial verification**. Approach every AI-generated legal claim with skepticism. Your role is not to confirm what the AI said, but to independently verify whether it's accurate, properly sourced, and appropriately disclaimed. When in doubt, verify. When you can't verify, flag it. Better to over-verify than to distribute inaccurate legal information.