blog-scraper
Scrape blog posts via RSS feeds (free, no API key) with Apify fallback for JS-heavy sites. Use when you need to monitor competitor blogs, track industry content, or aggregate blog posts by keyword.
Best use case
blog-scraper is best used when you need a repeatable AI agent workflow instead of a one-off prompt.
Scrape blog posts via RSS feeds (free, no API key) with Apify fallback for JS-heavy sites. Use when you need to monitor competitor blogs, track industry content, or aggregate blog posts by keyword.
Teams using blog-scraper should expect a more consistent output, faster repeated execution, less prompt rewriting.
When to use this skill
- You want a reusable workflow that can be run more than once with consistent structure.
When not to use this skill
- You only need a quick one-off answer and do not need a reusable workflow.
- You cannot install or maintain the underlying files, dependencies, or repository context.
Installation
Claude Code / Cursor / Codex
Manual Installation
- Download SKILL.md from GitHub
- Place it in
.claude/skills/blog-scraper/SKILL.mdinside your project - Restart your AI agent — it will auto-discover the skill
How blog-scraper Compares
| Feature / Agent | blog-scraper | Standard Approach |
|---|---|---|
| Platform Support | Not specified | Limited / Varies |
| Context Awareness | High | Baseline |
| Installation Complexity | Unknown | N/A |
Frequently Asked Questions
What does this skill do?
Scrape blog posts via RSS feeds (free, no API key) with Apify fallback for JS-heavy sites. Use when you need to monitor competitor blogs, track industry content, or aggregate blog posts by keyword.
Where can I find the source code?
You can find the source code on GitHub using the link provided at the top of the page.
SKILL.md Source
# Blog Scraper Scrape blog posts via RSS/Atom feeds (free) with optional Apify fallback for JS-heavy sites. ## Quick Start For RSS mode (free), only dependency is `pip install requests`. No API key needed. ```bash # Scrape a blog's RSS feed python3 skills/blog-scraper/scripts/scrape_blogs.py \ --urls "https://example.com/blog" --days 30 # Multiple blogs with keyword filter python3 skills/blog-scraper/scripts/scrape_blogs.py \ --urls "https://blog1.com,https://blog2.com" --keywords "AI,marketing" --output summary # Force Apify for JS-heavy sites python3 skills/blog-scraper/scripts/scrape_blogs.py \ --urls "https://example.com" --mode apify ``` ## How It Works ### Auto Mode (default) 1. For each URL, tries to discover an RSS/Atom feed: - Checks HTML `<link rel="alternate">` tags - Probes common paths: `/feed`, `/rss`, `/atom.xml`, `/feed.xml`, `/rss.xml`, `/blog/feed`, `/index.xml` 2. Parses discovered feeds (supports RSS 2.0 and Atom) 3. If any URLs fail, falls back to Apify `jupri/rss-xml-scraper` (if token available) 4. Applies date and keyword filtering client-side ### RSS Mode Only tries RSS feeds, no Apify fallback. ### Apify Mode Uses Apify actor directly, skipping RSS discovery. ## CLI Reference | Flag | Default | Description | |------|---------|-------------| | `--urls` | *required* | Blog URL(s), comma-separated | | `--keywords` | none | Keywords to filter (comma-separated, OR logic) | | `--days` | 30 | Only include posts from last N days | | `--max-posts` | 50 | Max posts to return | | `--mode` | auto | `auto` (RSS + fallback), `rss` (RSS only), `apify` (Apify only) | | `--output` | json | Output format: `json` or `summary` | | `--token` | env var | Apify token (only needed for Apify mode/fallback) | | `--timeout` | 300 | Max seconds for Apify run | ## Cost - **RSS mode:** Free (no API, no tokens) - **Apify mode:** Uses `jupri/rss-xml-scraper` — minimal Apify credits
Related Skills
web-archive-scraper
Search the Wayback Machine for archived versions of websites. Extract cached pages, customer lists, testimonials, and partner directories from sites that have changed or gone offline. Uses the free CDX API — no API key needed.
twitter-scraper
Search and scrape Twitter/X posts using Apify. Use when you need to find tweets, track brand mentions, monitor competitors on Twitter, or analyze Twitter discussions. Uses Twitter native search syntax (since:/until:) for reliable date filtering.
review-scraper
Scrape product reviews from G2, Capterra, and Trustpilot using Apify. Single script with platform dispatch. Use when you need to monitor competitor reviews, track product sentiment, or gather customer feedback from review sites.
reddit-scraper
Scrape and search Reddit posts using Apify. Use when you need to find Reddit discussions, track competitor mentions, monitor product feedback, discover pain points, or analyze subreddit content. Supports keyword filtering, time-based searches, and subreddit-specific queries.
product-hunt-scraper
Scrape Product Hunt trending products using Apify. Use when you need to discover new product launches, track competitors on Product Hunt, or monitor the startup ecosystem for relevant launches.
meta-ad-scraper
Scrape competitor ads from Meta's Ad Library (Facebook, Instagram, Messenger, Threads, WhatsApp). Search by company name, Facebook Page URL, or keyword. Returns ad creatives, spend estimates, reach, impressions, and campaign details. Use for competitive ad research, messaging analysis, and creative inspiration.
linkedin-profile-post-scraper
Scrape recent posts from LinkedIn profiles using Apify. Use when you need to monitor what specific people are posting on LinkedIn, track founder/exec activity, or gather LinkedIn content for competitive intelligence.
linkedin-job-scraper
Scrapes LinkedIn job postings using the JobSpy library (python-jobspy). Use this skill whenever the user wants to find jobs on LinkedIn, search for open roles, pull job listings, build a job pipeline, source job targets for GTM research, or monitor hiring signals. Even if the user just says "find me some jobs" or "what roles is [company] hiring for", use this skill. It runs a local Python script that outputs a CSV of job postings with title, company, location, salary, job type, description, and direct URLs.
hacker-news-scraper
Search Hacker News stories and comments using the free Algolia API. No Apify token needed. Use when you need to find HN discussions, track mentions, discover Show HN launches, or monitor tech community sentiment.
google-ad-scraper
Scrape competitor ads from Google's Ads Transparency Center (Search, YouTube, Display, Gmail). Search by company name, domain, or advertiser ID. Returns ad creatives, formats, targeting regions, and campaign details. Use for competitive ad research and messaging analysis.
conference-speaker-scraper
Extract speaker names, titles, companies, and bios from conference websites. Supports direct HTML scraping and Apify web scraper fallback for JS-heavy sites. Use for pre-event research and outreach targeting.
signal-detection-pipeline
Detect buying signals from multiple sources, qualify leads, and generate outreach context