stealth-browser

# Stealth Browser

Bypass bot detection and IP blocks with puppeteer-extra stealth plugin and optional Smartproxy residential proxy support.

## When to Use

- Websites blocking headless browsers or datacenter IPs
- Cloudflare/Vercel protection bypassing
- Sites detecting automation (Reddit, Twitter/X, signup flows, faucets)
- Protected content scraping
- Web automation requiring human-like behavior

## Tested Working On

✅ Relay.link (was blocked by Vercel, now works)
✅ X/Twitter profiles
✅ Bot detection tests (sannysoft.com)
✅ Faucet sites with protection
✅ Reddit (datacenter IP blocks)

## Quick Start

```bash
# Basic usage (stealth only)
node scripts/browser.js "https://example.com"

# With residential proxy (bypasses IP blocks)
node scripts/browser.js "https://example.com" --proxy

# Screenshot
node scripts/browser.js "https://example.com" --proxy --screenshot output.png

# Get HTML content
node scripts/browser.js "https://example.com" --proxy --html

# Get text content
node scripts/browser.js "https://example.com" --proxy --text
```

## Setup

### 1. Install Dependencies

```bash
cd /path/to/skill
npm install
```

Required packages (automatically handled by npm install with included package.json):
- `puppeteer-extra`
- `puppeteer-extra-plugin-stealth`
- `puppeteer`

### 2. Configure Proxy (Optional but Recommended)

For bypassing IP-based blocks, set up Smartproxy residential proxy:

Create `~/.config/smartproxy/proxy.json`:

```json
{
  "host": "proxy.smartproxy.net",
  "port": "3120",
  "username": "smart-ppz3iii4l2qr_area-US_life-30_session-xxxxx",
  "password": "your-password"
}
```

Get credentials from Smartproxy dashboard: https://dashboard.smartproxy.com

**Smartproxy session parameters:**
- `_area-US` → Use US residential IPs
- `_life-30` → Session lasts 30 minutes
- `_session-xxxxx` → Sticky session (same IP for duration)

Without proxy, the browser still uses stealth plugin to avoid detection, but may be blocked by IP-based protection.

## How It Works

### Stealth Features

The browser includes multiple anti-detection measures:

1. **puppeteer-extra-plugin-stealth**: Automatically applies all stealth evasions
   - Removes `navigator.webdriver` flag
   - Spoofs Chrome user agent and headers
   - Fakes plugins, languages, permissions
   - Removes automation signatures

2. **Human-like behaviors**:
   - Realistic viewport (1920x1080)
   - Updated user agent (Chrome 121)
   - Natural browser properties
   - No automation control flags

3. **Residential proxy** (when `--proxy` used):
   - Routes through residential IPs
   - Bypasses datacenter IP blocks
   - Sticky sessions (same IP per session)
   - Geographic targeting (US by default)

### Detection Bypass Comparison

| Protection | Headless Puppeteer | Stealth Plugin | + Residential Proxy |
|------------|-------------------|----------------|-------------------|
| navigator.webdriver | ❌ Detected | ✅ Hidden | ✅ Hidden |
| User Agent | ❌ Generic | ✅ Realistic | ✅ Realistic |
| WebGL/Canvas | ❌ Headless | ✅ Spoofed | ✅ Spoofed |
| IP Blocks | ❌ Datacenter | ❌ Datacenter | ✅ Residential |
| Cloudflare | ❌ Blocked | ⚠️  Sometimes | ✅ Usually works |
| Turnstile CAPTCHA | ❌ Blocked | ❌ Blocked | ⚠️  Reduced chance |

## Usage Examples

### Example 1: Check if Site Detects Automation

```bash
# Test on bot detection site
node scripts/browser.js "https://bot.sannysoft.com" --screenshot detection.png
```

Look for green checkmarks = undetected, red = detected.

### Example 2: Scrape Protected Page

```bash
# Get page text content
node scripts/browser.js "https://protected-site.com" --proxy --text > output.txt
```

### Example 3: Monitor Site Changes

```bash
# Take daily screenshot for comparison
node scripts/browser.js "https://target-site.com" --proxy --screenshot "$(date +%Y-%m-%d).png"
```

### Example 4: Extract Structured Data

```javascript
import { browse } from './scripts/browser.js';

const result = await browse('https://example.com', {
  proxy: true,
  html: true
});

// Parse result.html with cheerio or similar
console.log(result.html);
```

## Proxy Cost Considerations

**Smartproxy residential pricing:**
- ~$7.50/GB of traffic
- Average page load: 1-3 MB
- Rough cost: $0.01-0.03 per page

**When to use proxy:**
- Site explicitly blocks datacenter IPs (Reddit, some faucets)
- Cloudflare/Vercel protection detected
- Multiple requests from same IP getting rate-limited
- Geographic targeting needed (US vs international)

**When stealth-only is enough:**
- Site only checks for automation signatures, not IP
- Low-value scraping where IP blocks are acceptable
- Testing/development (proxy costs add up)

## Troubleshooting

### Browser Launch Fails

```
Error: Failed to launch the browser process
```

**Solution**: Install required system dependencies:

```bash
# Debian/Ubuntu
sudo apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 \
  libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 \
  libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 \
  libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
  libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 \
  libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation \
  libappindicator1 libnss3 lsb-release xdg-utils wget
```

### Proxy Authentication Fails

```
Error: net::ERR_PROXY_AUTH_REQUESTED
```

**Solution**: Check proxy credentials in `~/.config/smartproxy/proxy.json`. Verify username/password are correct in Smartproxy dashboard.

### Still Getting Detected

**Try these:**

1. **Update session ID** in proxy username (forces new IP):
   ```json
   "username": "smart-ppz3iii4l2qr_area-US_life-30_session-NEW_RANDOM_STRING"
   ```

2. **Increase wait time** before interacting with page:
   ```javascript
   await page.goto(url, { waitUntil: 'networkidle2' });
   await page.waitForTimeout(5000); // Wait 5s
   ```

3. **Check detection test**:
   ```bash
   node scripts/browser.js "https://bot.sannysoft.com" --proxy --screenshot test.png
   ```

4. **Try different geographic area** (if specific region is blocked):
   ```json
   "username": "smart-ppz3iii4l2qr_area-GB_life-30_session-xxxxx"
   ```

## Limitations

- **CAPTCHAs**: Stealth reduces but doesn't eliminate CAPTCHA challenges. For CAPTCHA solving, combine with 2captcha service.
- **JavaScript fingerprinting**: Advanced fingerprinting (Canvas, WebGL hash analysis) may still detect automation on highly protected sites.
- **Cost**: Residential proxy adds per-request cost. Use strategically.
- **Speed**: Proxy routing and stealth evasions add latency vs direct requests.

## Security Notes

**Capabilities:** This skill is **read-only** — it fetches web pages, captures screenshots, and extracts text/HTML. It does **not** perform any financial operations, value transfers, or wallet interactions.

**Authentication:** Proxy credentials are used solely for routing HTTP traffic through residential IPs. They do not grant access to any financial accounts or value-bearing systems.

- Proxy credentials contain sensitive auth tokens. Keep `~/.config/smartproxy/proxy.json` with 600 permissions.
- Never commit proxy credentials to git repositories.
- Residential proxy traffic is routed through real residential IPs. Respect rate limits and terms of service.
- No value-transfer risk: this tool cannot send transactions, move funds, or interact with smart contracts.

## See Also

- **2captcha skill**: For solving CAPTCHAs when stealth isn't enough
- **Smartproxy dashboard**: https://dashboard.smartproxy.com for usage monitoring
- **Bot detection test**: https://bot.sannysoft.com to verify stealth effectiveness