crawl-docs-skill
Run a Crawl4AI-based doc crawler and save internal pages as Markdown using page titles as filenames. Use when the user provides a docs URL and wants all internal subpages saved as .md files. Environment setup should only use uv.
Best use case
crawl-docs-skill is best used when you need a repeatable AI agent workflow instead of a one-off prompt.
Run a Crawl4AI-based doc crawler and save internal pages as Markdown using page titles as filenames. Use when the user provides a docs URL and wants all internal subpages saved as .md files. Environment setup should only use uv.
Teams using crawl-docs-skill should expect a more consistent output, faster repeated execution, less prompt rewriting.
When to use this skill
- You want a reusable workflow that can be run more than once with consistent structure.
When not to use this skill
- You only need a quick one-off answer and do not need a reusable workflow.
- You cannot install or maintain the underlying files, dependencies, or repository context.
Installation
Claude Code / Cursor / Codex
Manual Installation
- Download SKILL.md from GitHub
- Place it in
.claude/skills/crawl-docs-skill/SKILL.mdinside your project - Restart your AI agent — it will auto-discover the skill
How crawl-docs-skill Compares
| Feature / Agent | crawl-docs-skill | Standard Approach |
|---|---|---|
| Platform Support | Not specified | Limited / Varies |
| Context Awareness | High | Baseline |
| Installation Complexity | Unknown | N/A |
Frequently Asked Questions
What does this skill do?
Run a Crawl4AI-based doc crawler and save internal pages as Markdown using page titles as filenames. Use when the user provides a docs URL and wants all internal subpages saved as .md files. Environment setup should only use uv.
Where can I find the source code?
You can find the source code on GitHub using the link provided at the top of the page.
SKILL.md Source
# Crawl Docs Skill ## One-shot (Direct Run) - Run directly: `bash "<path-to-skill>/scripts/crawl_docs.sh" <URL> --out "$PWD/docs"` - Do NOT install dependencies or scaffold files first; run immediately. The script will bootstrap uv + venv only if needed. - Do NOT run tests or create plans; just execute the script and let it fail if the system is missing prerequisites (e.g. curl). - Default output dir (skill convention): `"$PWD/docs"` (absolute path to `./docs` under the current working directory); always pass it via `--out` unless the user specifies otherwise. - Optional flags: `--out <dir>`, `--max-pages <int>` ## Correct Usage (from real run) - First run: execute the one-shot command exactly once with no extra steps. - If it fails: fix the missing system prerequisite (most commonly `curl`) and re-run the same command. - Do not probe versions or add max-pages unless the user asks. ## Notes - Use page title (tab name) as the output filename; auto-deduplicate with numeric suffixes. - Default behavior has no max pages unless `--max-pages` is provided.
Related Skills
google-docs-manager
Expert in Google Docs management. Use when creating, reading, updating, formatting, or managing Google Docs with markdown support, advanced formatting, tables with full manipulation, images with styling, lists, headers/footers, and table of contents.
genesis-tools:living-docs
Self-maintaining documentation system. Bootstraps, validates, refines, and optimizes codebase documentation. Creates minimal, token-efficient doc chunks. Use when creating, updating, or auditing project documentation.
generate-docstrings
Create docstrings for functions and classes. Use when documenting code APIs.
generate-agent-docs
Generates documentation and usage guides for agents, skills, prompts, and instructions. Works with GitHub Copilot, Claude Code, Codex, OpenCode, and other providers. Use when onboarding team members, creating README files for your customizations, or generating usage examples for existing agents.
feature-docs
[Documentation] Create or update business feature documentation in docs/business-features/{Module}/. Generates comprehensive 26-section docs with verified code evidence and AI companion files. Triggers on: feature docs, business feature documentation, module documentation, document feature, update feature docs, ai companion, ai context file, quick feature docs, feature readme, single file docs, verified documentation.
faf-docs
Access FAF documentation, guides, and resources. Answers questions about The Reading Order, IANA registration, Podium scoring, format specification, and best practices. Use when user asks "how does FAF work", "show me docs", "explain The Reading Order", or needs reference information.
explanation-docs
Explanation documentation patterns for understanding-oriented content - conceptual guides that explain why things work the way they do
docsumo-automation
Automate Docsumo tasks via Rube MCP (Composio). Always search tools first for current schemas.
docs
Generates documentation files including NotebookLM YAML and slide content. Use when user mentions ドキュメント, document, YAML, NotebookLM, スライド, slide, プレゼン. Do NOT load for: 実装作業, コード修正, レビュー, デプロイ.
docs-verify
Validates documentation quality and freshness — checks for broken links, stale content, llms.txt sync, image issues, heading hierarchy, and badge URLs. Runs locally or in CI. Use to catch documentation decay before it reaches users.
docs-sync
Sync AILANG documentation website with codebase reality. Use after releases, when features are implemented, or when website accuracy is questioned. Checks design docs vs website, validates examples, updates feature status.
docs-plugin
Command: docs-plugin