regex-patterns

# regex-patterns skill

## Pattern structure

Every pattern in `src/patterns.ts` implements the `Pattern` interface:

```typescript
interface Pattern {
  id:       string;         // unique identifier, SCREAMING_SNAKE_CASE
  name:     string;         // human-readable name for output
  regex:    RegExp;         // detection regex
  severity: Severity;       // critical | high | medium | low
  entropy?: number;         // minimum Shannon entropy to fire (optional)
}
```

## Complete pattern list with rationale

```typescript
// src/patterns.ts
import type { Pattern } from './types.js';

export const PATTERNS: Pattern[] = [

  // --- AWS ---
  {
    id:       'AWS_ACCESS_KEY',
    name:     'AWS Access Key ID',
    regex:    /\bAKIA[0-9A-Z]{16}\b/,
    severity: 'critical',
    // No entropy check: format is highly specific, false positives are rare
  },
  {
    id:       'AWS_SECRET_KEY',
    name:     'AWS Secret Access Key',
    regex:    /\b(?:aws_secret(?:_access)?_key|AWS_SECRET(?:_ACCESS)?_KEY)\s*[:=]\s*['"]?([A-Za-z0-9/+]{40})\b/i,
    severity: 'critical',
    entropy:  4.5,
    // Entropy required: 40-char base64 can match version strings
  },

  // --- GitHub ---
  {
    id:       'GITHUB_TOKEN',
    name:     'GitHub Personal Access Token',
    regex:    /\b(ghp_[A-Za-z0-9]{36}|gho_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{82})\b/,
    severity: 'critical',
  },

  // --- Slack ---
  {
    id:       'SLACK_TOKEN',
    name:     'Slack API Token',
    regex:    /\bxox[baprs]-[0-9A-Za-z\-]{10,}\b/,
    severity: 'high',
  },

  // --- Private keys ---
  {
    id:       'PRIVATE_KEY_HEADER',
    name:     'PEM Private Key',
    regex:    /-----BEGIN\s+(?:RSA\s+|EC\s+|DSA\s+|OPENSSH\s+|ENCRYPTED\s+)?PRIVATE KEY-----/,
    severity: 'critical',
  },

  // --- Database connection strings ---
  {
    id:       'CONNECTION_STRING',
    name:     'Database Connection String',
    regex:    /(?:postgres(?:ql)?|mysql|mongodb(?:\+srv)?|redis):\/\/[^@\s:]+:[^@\s]+@[^\s'"]+/i,
    severity: 'high',
    // Matches: protocol://user:password@host/db
  },

  // --- Generic ---
  {
    id:       'GENERIC_API_KEY',
    name:     'Generic API Key Assignment',
    regex:    /\b(?:api[_-]?key|api[_-]?secret|apikey|api_token)\s*[:=]\s*['"]?([A-Za-z0-9_\-]{20,})/i,
    severity: 'medium',
    entropy:  3.5,
  },

  // --- JWT ---
  {
    id:       'JWT_TOKEN',
    name:     'JSON Web Token',
    regex:    /\bey[A-Za-z0-9_\-]{10,}\.[A-Za-z0-9_\-]{10,}\.[A-Za-z0-9_\-]{10,}\b/,
    severity: 'high',
    // eyJ... is the base64url encoding of {"...}
  },

  // --- High-entropy catch-all ---
  {
    id:       'HIGH_ENTROPY_STRING',
    name:     'High-entropy string (possible secret)',
    regex:    /['"]([A-Za-z0-9+/=_\-]{20,})['"]/,
    severity: 'low',
    entropy:  4.5,
    // Last resort: anything in quotes that is long and high-entropy
  },
];
```

## Shannon entropy implementation

```typescript
// src/entropy.ts
export function shannonEntropy(s: string): number {
  if (s.length === 0) return 0;
  const freq = new Map<string, number>();
  for (const c of s) {
    freq.set(c, (freq.get(c) ?? 0) + 1);
  }
  let entropy = 0;
  for (const count of freq.values()) {
    const p = count / s.length;
    entropy -= p * Math.log2(p);
  }
  return entropy;
}
```

Expected values:
| Input | Expected entropy |
|-------|-----------------|
| `"aaaa"` | 0.0 |
| `"ab"` | 1.0 |
| `"abcd"` | 2.0 |
| `"abcdefgh"` | 3.0 |
| `"AKIAIOSFODNN7EXAMPLE"` | ~3.7 |
| `"wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"` | ~5.2 |

## Pattern matching in scanner

The scanner applies each pattern to each line of each file. When using capture groups (group 1), entropy is evaluated on the captured group rather than the full match:

```typescript
const match = pattern.regex.exec(line);
if (!match) continue;

// Use capture group 1 if present (more specific), else full match
const value = match[1] ?? match[0];

if (pattern.entropy !== undefined) {
  const e = shannonEntropy(value);
  if (e < pattern.entropy) continue;  // entropy too low, skip
}
```

## Adding a custom pattern

1. Add to `customPatterns` in `.secretscanrc.json`:
```json
{
  "customPatterns": [{
    "id": "MY_TOKEN",
    "name": "My service token",
    "regex": "svc_[A-Za-z0-9]{32}",
    "severity": "critical",
    "entropy": 3.8
  }]
}
```

2. The scanner loads this at startup and compiles `regex` string to `RegExp`. Invalid regex strings produce a startup error.

## Pattern testing checklist

For each pattern, test:
1. A known-real secret value matches
2. A similar-looking non-secret does not match
3. The false-positive rate on a corpus of typical TypeScript/JavaScript source files is acceptable
4. The entropy filter (if set) correctly rejects low-entropy strings that match the regex

## Common false-positive sources

| Pattern | Common FP | Fix |
|---------|-----------|-----|
| GENERIC_API_KEY | Version strings ("api_key": "v1.2.3-rc1") | Entropy filter >= 3.5 |
| HIGH_ENTROPY_STRING | Base64 image data | Entropy filter >= 4.5 |
| CONNECTION_STRING | Test URLs with `example.com` | Add to .secretscanignore |
| JWT_TOKEN | Fixture JWTs in test files | Add test dir to .secretscanignore |