edstem

# EdStem

Fetch and organize EdStem discussion threads from any course or institution with automatic staff/student differentiation.

## Quick Start

Fetch recent threads for any course:

```bash
cd /home/axel/.openclaw/workspace/skills/edstem/scripts
python3 fetch-edstem.py <course_id> [output_dir] [--course-name "Course Name"]
```

**Examples:**
```bash
# Fetch to default directory (./edstem-<course_id>)
python3 fetch-edstem.py 92041

# Fetch to specific directory
python3 fetch-edstem.py 92041 ./machine-learning

# Specify course name for clearer output
python3 fetch-edstem.py 92041 --course-name "Machine Learning"

# Combine directory and course name
python3 fetch-edstem.py 92041 ./ml-course --course-name "Machine Learning"

# Fetch more threads (default is 10)
python3 fetch-edstem.py 92041 --limit 25
```

## Finding Your Course ID

To find your EdStem course ID:

1. Log into EdStem and navigate to your course
2. Look at the URL: `https://edstem.org/us/courses/<course_id>/`
3. The number in the URL is your course ID

Alternatively, use the API to list your courses:
```bash
curl -H "Authorization: Bearer YOUR_TOKEN" https://us.edstem.org/api/user | jq '.courses[] | {id: .course.id, name: .course.name}'
```

## What Gets Fetched

For each course:
- **threads.json** - Full thread list with metadata
- **thread-XXX.md** - Individual threads formatted as markdown
  - Thread title, category, timestamps
  - Original post content
  - All answers and comments
  - **[STAFF]** or **[STUDENT]** tags on every post

## Features

- **Institution-agnostic**: Works with any school using EdStem
- **Staff differentiation**: Clearly marks instructor/TA posts vs student posts
- **Structured output**: Markdown format for easy reading and searching
- **API-based**: Uses EdStem's official API (no scraping)
- **Flexible output**: Choose your own output directory and organization scheme

## Authentication

The skill uses a bearer token stored in the Python script. To use with your own account:

1. Log into EdStem in your browser
2. Open Developer Tools → Network tab
3. Reload any EdStem page
4. Find an API request and copy the `Authorization: Bearer ...` token
5. Update `ED_TOKEN` in `scripts/fetch-edstem.py`

**Current token location:** Line 20 in `scripts/fetch-edstem.py`

If API calls fail (401 Unauthorized), your token likely expired and needs refresh.

## Scripts

### fetch-edstem.py (recommended)
Full-featured Python script with markdown formatting and staff/student differentiation.

**Usage:**
```bash
python3 scripts/fetch-edstem.py <course_id> [output_dir] [options]
```

**Options:**
- `output_dir` - Where to save threads (default: `./edstem-<course_id>`)
- `--course-name NAME` - Display name for the course
- `--limit N` - Number of threads to fetch (default: 10)

**Features:**
- Fetches thread metadata and full details
- Full markdown formatting with answers and comments
- Automatic staff role detection
- JSON cache of thread list
- Auto-creates output directory

### fetch-edstem.sh (lightweight alternative)
Bash/curl version for raw JSON fetching without dependencies.

**Usage:**
```bash
bash scripts/fetch-edstem.sh <course_id> [output_dir]
```

**Outputs:**
- Raw JSON files for each thread
- Requires manual formatting or post-processing

## Common Workflows

### Check for new posts
```bash
python3 scripts/fetch-edstem.py 92041 ~/courses/ml-spring-2025
```

### Sync multiple courses
```bash
# Create a simple sync script
for course in "92041:machine-learning" "94832:advanced-rl"; do
    IFS=':' read -r id name <<< "$course"
    python3 scripts/fetch-edstem.py $id ~/courses/$name --course-name "$name"
done
```

### Review recent activity
After fetching, check the markdown files:
```bash
ls -lt ./edstem-92041/*.md | head
cat ./edstem-92041/thread-001.md
```

### Search across threads
```bash
grep -r "gradient descent" ./edstem-92041/*.md
```

## Output Structure

```
<output_dir>/
├── threads.json              # Thread metadata
├── thread-001.md             # Individual threads
├── thread-002.md
└── ...
```

Each markdown file contains:
- Thread metadata (number, title, category, timestamps)
- Original post with author role
- All answers (sorted, with role tags)
- All comments (with role tags)

## Integration Examples

### With LLM agents
```bash
# Fetch threads and analyze with your agent
python3 fetch-edstem.py 92041 ./course-data
# Then: "Summarize the most common questions in ./course-data/"
```

### Automated monitoring
```bash
# Add to cron for daily sync
0 9 * * * cd /path/to/skills/edstem/scripts && python3 fetch-edstem.py 92041 ~/courses/ml
```

### Custom organization
```bash
# Organize by semester and institution
python3 fetch-edstem.py 92041 ~/school/stanford/2025-spring/cs229
python3 fetch-edstem.py 94832 ~/school/mit/2025-spring/6.7920
```

## Troubleshooting

**401 Unauthorized:** Token expired. Re-authenticate and update `ED_TOKEN` in the script.

**Course not found:** Verify the course ID and that your account has access.

**Empty threads:** Check that the course has discussion posts and you're enrolled.

**Rate limiting:** EdStem may rate-limit API requests. Add delays between fetches if needed.

## Contributing

This skill is open-source and institution-agnostic by design. Improvements welcome:
- Better content parsing (EdStem uses XML-based document format)
- Support for filtering by category or date range
- Incremental sync (only fetch new threads)
- Export to other formats (JSON, HTML, etc.)

## Version History

- **1.1.0** - Made institution-agnostic with flexible parameters
- **1.0.0** - Initial release