twitter-search

# Twitter Search and Analysis

## Overview

Search Twitter for keywords using advanced search syntax, fetch up to 1000 relevant tweets, and analyze the data to produce professional reports with insights, statistics, and actionable recommendations.

## Prerequisites

**API Key Required**: Users must configure their Twitter API key from https://twitterapi.io

The API key can be provided in three ways:
1. **Environment variable** (recommended): Set `TWITTER_API_KEY` in your `~/.bashrc` or `~/.zshrc`
   ```bash
   echo 'export TWITTER_API_KEY="your_key_here"' >> ~/.bashrc
   source ~/.bashrc
   ```
2. **As an argument**: Use `--api-key YOUR_KEY` with the wrapper script
3. **Passed directly**: As first argument to the Python script

## Quick Start

### Using the Wrapper Script (Recommended)

The wrapper script automatically handles environment variable loading and dependency checks:

```bash
# Basic search (uses TWITTER_API_KEY from shell config)
./scripts/run_search.sh "AI"

# With custom API key
./scripts/run_search.sh "AI" --api-key YOUR_KEY

# With options
./scripts/run_search.sh "\"Claude AI\"" --max-results 100 --format summary

# Advanced query
./scripts/run_search.sh "from:elonmusk since:2024-01-01" --query-type Latest
```

### Direct Python Script Usage

```bash
# Search for a keyword
scripts/twitter_search.py "$API_KEY" "AI"

# Search with multiple keywords
scripts/twitter_search.py "$API_KEY" "\"ChatGPT\" OR \"Claude AI\""

# Search from specific user
scripts/twitter_search.py "$API_KEY" "from:elonmusk"

# Search with date range
scripts/twitter_search.py "$API_KEY" "Bitcoin since:2024-01-01"
```

### Advanced Queries

```bash
# Complex query: AI tweets from verified users, English only
scripts/twitter_search.py "$API_KEY" "AI OR \"machine learning\" lang:en filter:verified"

# Recent crypto tweets with minimum engagement
scripts/twitter_search.py "$API_KEY" "Bitcoin min_retweets:10 lang:en"

# From specific influencers
scripts/twitter_search.py "$API_KEY" "from:elonmusk OR from:VitalikButerin since:2024-01-01"
```

### Output Format

```bash
# Full JSON with all tweets
scripts/twitter_search.py "$API_KEY" "AI" --format json

# Summary with statistics (default)
scripts/twitter_search.py "$API_KEY" "AI" --format summary
```

### Options

- `--max-results N`: Maximum tweets to fetch (default: 1000)
- `--query-type Latest|Top`: Sort order (default: Top for relevance)
- `--format json|summary`: Output format (default: summary)

## Workflow

### 1. Understand User Requirements

Clarify the analysis goal:
- What topic/keyword to search?
- Date range preference?
- Specific users to include/exclude?
- Language preference?
- Type of insights needed (trends, sentiment, influencers)?

### 2. Build the Search Query

Use [Twitter Advanced Search](https://github.com/igorbrigadir/twitter-advanced-search) syntax:

| Syntax | Example | Description |
|--------|---------|-------------|
| `keyword` | `AI` | Single keyword |
| `"phrase"` | `"machine learning"` | Exact phrase |
| `OR` | `AI OR ChatGPT` | Either term |
| `from:user` | `from:elonmusk` | From specific user |
| `to:user` | `to:elonmusk` | Reply to user |
| `since:DATE` | `since:2024-01-01` | After date |
| `until:DATE` | `until:2024-12-31` | Before date |
| `lang:xx` | `lang:en` | Language code |
| `#hashtag` | `#AI` | Hashtag |
| `filter:links` | `filter:links` | Tweets with links |
| `min_retweets:N` | `min_retweets:100` | Minimum retweets |

### 3. Fetch Data

Execute the search script:

```bash
scripts/twitter_search.py "$API_KEY" "YOUR_QUERY" --max-results 1000 --query-type Top
```

**Important**: Default is 1000 tweets maximum. The script automatically:
- Paginates through all available results
- Stops at 1000 tweets (API limit consideration)
- Handles errors gracefully

### 4. Analyze and Generate Report

After fetching data, produce a comprehensive professional report with:

#### Report Structure

1. **Executive Summary** (2-3 sentences)
   - What was searched
   - Key findings overview

2. **Data Overview**
   - Total tweets analyzed
   - Date range of data
   - Query parameters used

3. **Key Metrics**
   - Total engagement (likes, retweets, replies, quotes, views)
   - Average engagement per tweet
   - Language distribution
   - Reply vs. original tweet ratio

4. **Top Content Analysis**
   - Most retweeted tweets (with **URL links** to original tweets)
   - Most liked tweets (with **URL links** to original tweets)
   - Top hashtags with frequency
   - Most mentioned users
   - Selected tweet examples with full URL references

5. **Influencer Analysis**
   - Top users by follower count
   - Most active users
   - Verified user percentage

6. **Trend Insights** (based on data patterns)
   - Emerging themes
   - Sentiment indicators
   - Temporal patterns
   - Conversation drivers

7. **Key Takeaways**
   - 3-5 bullet points of core insights
   - Data-backed conclusions

8. **Actionable Recommendations**
   - Specific, implementable suggestions
   - Based on the data findings
   - Prioritized by impact

#### Analysis Guidelines

- **Be data-driven**: Every claim should reference actual metrics
- **Provide context**: Explain why metrics matter
- **Identify patterns**: Look for trends across the dataset
- **Stay objective**: Present facts, avoid speculation
- **Be specific**: Recommendations should be concrete and actionable
- **Consider external context**: Use web search for background when relevant

### 5. Output Format

Present the report in clear markdown with:
- Headers for each section
- Tables for structured data
- Bullet points for lists
- Bold for key metrics
- Code blocks for tweet examples
- **Clickable URLs** for all referenced tweets (format: `[@username](https://x.com/username/status/tweet_id)`)

#### Tweet URL Format

Always include clickable links to tweets:
```markdown
| Author | Tweet | URL |
|--------|-------|-----|
| @user | Summary of tweet content | [View](https://x.com/user/status/123456) |
```

Or inline format:
```markdown
- **@username**: Tweet summary - [View Tweet](https://x.com/username/status/123456)
```

## Query Examples by Use Case

### Trend Analysis
```
"AI" OR "artificial intelligence" lang:en min_retweets:50
```

### Competitor Monitoring
```
from:competitor1 OR from:competitor2 since:2024-01-01
```

### Product Launch Tracking
```
#ProductName OR "Product Name" lang:en filter:verified
```

### Crisis Monitoring
```
#BrandName OR "Brand Name" lang:en --query-type Latest
```

### Influencer Discovery
```
#Topic lang:en min_retweets:100 min_faves:500
```

### Sentiment Analysis
```
"brand name" OR #BrandName lang:en --max-results 1000
```

## Resources

### scripts/run_search.sh (Wrapper Script)

Convenience wrapper that handles environment variable loading and dependency checks:
- Automatically loads `TWITTER_API_KEY` from `~/.bashrc` or `~/.zshrc`
- Checks Python availability and installs missing dependencies
- Provides user-friendly error messages
- Supports all command-line options from the Python script

**Usage**:
```bash
./scripts/run_search.sh <query> [options]
```

**Options**:
- `--api-key KEY`: Override environment variable API key
- `--max-results N`: Maximum tweets to fetch (default: 1000)
- `--query-type Latest|Top`: Sort order (default: Top)
- `--format json|summary`: Output format (default: json)

### scripts/twitter_search.py

Executable Python script that:
- Fetches tweets from Twitter API
- Handles pagination automatically
- Extracts key tweet metrics
- Calculates aggregate statistics
- Outputs structured JSON data

**Usage**:
```bash
scripts/twitter_search.py <api_key> <query> [options]
```

### references/twitter_api.md

Comprehensive API documentation including:
- Complete parameter reference
- Query syntax guide
- Response structure details
- Pagination instructions
- Best practices for analysis
- Error handling guide

**Read this when**: Building complex queries or understanding data structure.

## Tips for Better Analysis

1. **Use Top query type** for trend analysis (more relevant results)
2. **Set date filters** for timely insights
3. **Filter by language** for accurate text analysis
4. **Include minimum engagement** to filter noise
5. **Combine with web search** to validate trends
6. **Look beyond metrics** - analyze content themes
7. **Track hashtags** to identify sub-conversations
8. **Identify influencers** by combining followers + engagement

## Error Handling

If the script fails:
- Check API key validity
- Verify query syntax
- Ensure network connectivity
- Check rate limits (if applicable)
- Review error messages for specific issues