skroller

# Skroller - Social Media Content collection

Automate the collection and analysis of publicly available social media content. This skill handles content collection, filtering, and export with compliance safeguards.

## ⚖️ Legal & Compliance Requirements

**Before using this skill:**
1. **Review Platform ToS** - Each platform has different rules about automated access
2. **Check robots.txt** - Respect disallowed paths
3. **Rate Limiting** - Stay within platform rate limits to avoid service disruption
4. **Privacy Laws** - GDPR, CCPA, and other regulations apply to personal data
5. **Permitted Use** - Research, personal analysis, competitive intelligence (where allowed)
6. **Prohibited Use** - Spam, harassment, data resale, manipulation, bypassing auth

**Data Protection:**
- Anonymize personal data when storing
- Honor deletion requests (GDPR Art. 17)
- Limit retention to necessary periods
- Document lawful basis for processing
- Do not scrape sensitive personal data

## Core Capabilities

- **Content collection** - Gather publicly available posts from platforms
- **Data extraction** - Pull text, timestamps, engagement metrics, author info
- **Smart filtering** - Filter by keywords, hashtags, date ranges, engagement thresholds
- **Deduplication** - Track seen posts to avoid duplicates across sessions
- **Export formats** - JSON, CSV, Markdown, or direct to note apps (Bear/Obsidian)
- **Rate limiting** - Respect platform constraints and avoid service disruption

## Platform Support

| Platform | Approach | Notes |
|----------|----------|-------|
| Twitter/X | Browser automation | Use Playwright; handle login if needed |
| Instagram | Browser automation | Rate-limited; use sparingly |
| TikTok | Browser automation | Heavy JS; may need longer waits |
| Reddit | API + browser | Prefer API where possible |
| LinkedIn | Browser automation | Login required for most content |
| YouTube | API + browser | Comments via browser, videos via API |
| Product Hunt | Browser automation | Product discovery, launches |
| Medium | Browser automation | Articles, blog posts |
| GitHub | Browser automation | Issues, discussions, repos |
| Pinterest | Browser automation | Visual content, pins |

## Quick Start

### Basic Scroll and Extract

```bash
# Scroll Twitter feed, extract 50 posts about "AI"
node scripts/skroller.js --platform twitter --query "AI" --limit 50 --output posts.json
```

### Monitor Brand Mentions

```bash
# Monitor Reddit for brand mentions, export to Markdown
node scripts/skroller.js --platform reddit --query "mybrand" --format markdown --output mentions.md
```

### Competitive Research

```bash
# Scroll competitor Instagram, capture top posts by engagement
node scripts/skroller.js --platform instagram --profile @competitor --min-likes 1000 --output competitor-posts.json
```

## Scripts

### `scripts/skroller.js` - Main scrolling engine

JavaScript script using Playwright for browser automation.

```bash
# Using npm scripts (recommended)
npm run scroll -- --platform twitter --query "AI" --limit 50 --output posts.json

# Direct execution
node scripts/skroller.js --platform twitter --query "AI" --limit 50 --output posts.json
```

**Options:**
- `--platform` - Target platform (required): twitter, reddit, instagram, tiktok, linkedin, youtube, producthunt, medium, github, pinterest
- `--query` - Search keyword/hashtag
- `--profile` - Specific profile to scroll
- `--limit` - Max posts to scrape (default: 50)
- `--min-likes` - Filter by minimum engagement
- `--format` - Output format: json, csv, markdown (default: json)
- `--output` - Output file path
- `--screenshot` - Capture screenshots for debugging
- `--dedupe` - Skip previously seen posts

### `scripts/feed-digest.js` - Generate digests

Creates summary digests from exported post data.

```bash
npm run digest -- --input posts.json --output digest.md
# or: node scripts/feed-digest.js --input posts.json --output digest.md
```

### `scripts/export-to-notes.js` - Unified note app exporter

Exports scraped posts to multiple note applications with a single command.

**Supported apps:** Bear, Obsidian, Notion, Apple Notes, Evernote, OneNote, Google Keep, Roam Research, Logseq, Anytype

```bash
# Using npm scripts (recommended)
npm run export -- --input posts.json --app obsidian --vault ~/Documents/Obsidian

# Direct execution
node scripts/export-to-notes.js --input posts.json --app bear --tags "ai,research"
node scripts/export-to-notes.js --input posts.json --app notion --api-key $NOTION_TOKEN
node scripts/export-to-notes.js --input posts.json --app apple-notes
node scripts/export-to-notes.js --input posts.json --app evernote --output export.enex
node scripts/export-to-notes.js --input posts.json --app one-note --access-token $MS_TOKEN
node scripts/export-to-notes.js --input posts.json --app keep --output keep.html
node scripts/export-to-notes.js --input posts.json --app roam --output roam.md
node scripts/export-to-notes.js --input posts.json --app logseq --vault ~/Documents/Logseq
node scripts/export-to-notes.js --input posts.json --app anytype --output anytype.md
node scripts/export-to-notes.js --input posts.json --app obsidian --dry-run
```

**Configuration:** Set defaults in `.skroller-config.json`:
```json
{
  "export": {
    "defaultApp": "obsidian",
    "vault": "~/Documents/Obsidian",
    "folder": "Skroller",
    "notionDatabaseId": "<db-id>"
  }
}
```

**Requirements by app:**
- **Bear:** grizzly CLI (`go install github.com/tylerwince/grizzly/cmd/grizzly@latest`)
- **Obsidian:** Vault path
- **Notion:** API key (create at notion.so)
- **Apple Notes:** macOS with Notes app
- **Evernote:** Manual ENEX import
- **OneNote:** Microsoft Graph access token
- **Google Keep:** Manual HTML import
- **Roam Research:** Markdown import (drag MDL file into Roam)
- **Logseq:** Vault path (writes to pages/ folder)
- **Anytype:** Markdown import (use Anytype Import feature)

## Configuration

### `.skroller-config.json`

Store default settings:

```json
{
  "defaultLimit": 50,
  "scrollDelayMs": 1500,
  "userAgent": "Mozilla/5.0 ...",
  "platforms": {
    "twitter": {
      "loginRequired": false,
      "rateLimit": "100 requests/hour"
    },
    "instagram": {
      "loginRequired": true,
      "rateLimit": "50 requests/hour"
    }
  },
  "export": {
    "defaultFormat": "json",
    "includeImages": true,
    "includeMetrics": true
  }
}
```

### Authentication

Some platforms require login. Store credentials securely:

```bash
# For platforms requiring auth, set environment variables
export SKROLLR_TWITTER_COOKIE="<auth cookie>"
export SKROLLR_INSTAGRAM_USER="<username>"
export SKROLLR_INSTAGRAM_PASS="<password>"
```

**Security note:** Never commit auth files. Use `.env` with `.gitignore`.

## Output Structure

### JSON Output (default)

```json
{
  "platform": "twitter",
  "query": "AI",
  "scrapedAt": "2026-03-14T20:30:00Z",
  "posts": [
    {
      "id": "1234567890",
      "author": "@username",
      "text": "Post content here...",
      "timestamp": "2026-03-14T18:00:00Z",
      "likes": 150,
      "retweets": 42,
      "replies": 12,
      "url": "https://twitter.com/...",
      "media": ["image1.jpg"],
      "hashtags": ["#AI", "#ML"]
    }
  ]
}
```

### Markdown Output

```markdown
## Twitter Posts: "AI" (2026-03-14)

### @username - 150 likes
Post content here...

[View post](https://twitter.com/...)

---
```

## Filtering Strategies

### Keyword Matching
- Exact match: `"exact phrase"`
- Boolean: `AI AND (startup OR venture)`
- Exclude: `AI -crypto`

### Engagement Thresholds
- Filter low-quality: `--min-likes 100`
- Viral content: `--min-shares 500`

### Time Windows
- Recent only: `--date-from 2026-03-14`
- Historical: `--date-from 2026-01-01 --date-to 2026-01-31`

## Best Practices

### Rate Limiting
- Add delays between scrolls: `--delay 2000`
- Respect platform limits (see config)
- Use proxies for high-volume scraping

### Content Quality
- Filter by engagement to find signal
- Dedupe across sessions
- Export with timestamps for freshness tracking

### Ethics & Compliance
- Check platform ToS before scraping
- Don't scrape personal data at scale
- Use for research/curation, not spam

## Troubleshooting

### Scroll stops early
- Increase `--limit` or check for login requirements
- Some platforms detect automation; add random delays

### Missing content
- Some platforms lazy-load; increase scroll delay
- Try `--screenshot` to debug what's visible

### Rate limited
- Reduce frequency; use `--delay`
- Check platform-specific rate limits in config

## Integration with Other Skills

- **bear-notes**: Export via `export-to-notes.js --app bear`
- **obsidian**: Export via `export-to-notes.js --app obsidian`
- **notion**: Export via `export-to-notes.js --app notion`
- **github**: Create issues from scraped feedback/mentions

```bash
# Scroll and export to Obsidian in one command
node scripts/skroller.js --platform twitter --query "AI" --limit 20 --output ai.json && \
  node scripts/export-to-notes.js --input ai.json --app obsidian --vault ~/Documents/Obsidian

# Scroll and export to Notion
node scripts/skroller.js --platform reddit --query "startups" --limit 30 --output startups.json && \
  node scripts/export-to-notes.js --input startups.json --app notion --api-key $NOTION_TOKEN

# Use npm scripts for cleaner commands
npm run scroll -- --platform twitter --query "tech" --limit 25 --output tech.json
npm run export -- --input tech.json --app bear --tags "tech,research"
```

## See Also

- `references/platform-details.md` - Platform-specific selectors and quirks
- `references/rate-limits.md` - Rate limit guidelines per platform
- `assets/selector-cheatsheet.md` - CSS selectors for each platform