obsidian-conversation-backup

# Obsidian Conversation Backup

Automatically backs up Clawdbot conversations to Obsidian with beautiful chat-style formatting. Prevents data loss from `/new` resets with hourly incremental snapshots.

## Features

- **Incremental backups**: Hourly snapshots of new messages only (no duplication)
- **Chat formatting**: Obsidian callouts with emojis, timestamps, proper multi-paragraph support
- **Hourly breakdowns**: Organize conversations by clock hour for easy reference
- **Zero token cost**: Pure shell scripting, no LLM calls
- **Smart filtering**: Skips empty messages and system notifications

## Quick Setup

### Installation

```bash
# Extract the skill (if downloaded as .skill file)
unzip obsidian-conversation-backup.skill
cd obsidian-conversation-backup

# Run installer (interactive)
chmod +x install.sh
./install.sh
```

The installer will ask for:
- Obsidian vault path
- Session directory location
- Tracking files location

**Or manual setup:**

1. Copy `config.example` to `config`
2. Edit `config` with your paths
3. Make scripts executable: `chmod +x scripts/*.sh`

### Enable Automatic Backups

Add to crontab for hourly backups:

```bash
crontab -e

# Add this line (runs every hour at :00)
0 * * * * /path/to/obsidian-conversation-backup/scripts/monitor_and_save.sh
```

### Customize Chat Appearance (Optional)

Edit `scripts/format_message_v2.jq` to change:
- User emoji (default: 🐉)
- Assistant emoji (default: 🦞)  
- Callout types (default: `[!quote]` for user, `[!check]` for assistant)

## Usage

### Automatic Incremental Backups

Once configured in cron, the system runs automatically:

**Every hour:**
- Checks for new messages (≥10 lines)
- Creates incremental snapshot if found
- Saves to: `YYYY-MM-DD-HHmm-incremental.md`
- Skips if no new conversation

**Example output:**
```
2026-01-20-1500-incremental.md (messages from last save to now)
2026-01-20-1600-incremental.md (new messages since 15:00)
2026-01-20-1700-incremental.md (new messages since 16:00)
```

**Protection:** Max conversation loss = 1 hour

### On-Demand Full Snapshot

Save complete conversation anytime:

```bash
scripts/save_full_snapshot.sh [topic-name]
```

**Examples:**
```bash
scripts/save_full_snapshot.sh important-decisions
scripts/save_full_snapshot.sh bug-fix-discussion
scripts/save_full_snapshot.sh  # uses "full-conversation" as default
```

### Hourly Breakdown (Organization)

Create organized breakdown by clock hour:

```bash
scripts/create_hourly_snapshots.sh YYYY-MM-DD
```

**Example:**
```bash
scripts/create_hourly_snapshots.sh 2026-01-20
```

**Output:**
```
2026-01-20-1500-hourly.md (15:00-15:59 messages)
2026-01-20-1600-hourly.md (16:00-16:59 messages)
2026-01-20-1700-hourly.md (17:00-17:59 messages)
```

**Use case:** End-of-day organization for easy reference

## Chat Format

Messages appear as colored Obsidian callouts:

**User messages** (blue `[!quote]` callout):
```
> [!quote] 🐉 User · 15:30
> This is my message
```

**Assistant messages** (green `[!check]` callout):
```
> [!check] 🦞 Zoidbot · 15:31  
> This is the response
```

**Features:**
- Timestamps (HH:MM format)
- Multi-paragraph support (uses `<br><br>` for paragraph breaks)
- Proper line wrapping (all lines prefixed with `> `)
- Empty messages filtered out
- System notifications excluded

## Token Monitoring

The `monitor_and_save.sh` script also tracks token usage:

**Warnings via Telegram:**
- **800k tokens (80%)**: "Consider /new soon"
- **900k tokens (90%)**: "Run /new NOW"

**Implementation:**
```bash
# Sends warning only when crossing threshold (one-time)
# No repeated warnings
# Resets when back under 800k
```

## File Structure

```
scripts/
├── monitor_and_save.sh           # Hourly incremental backup + token monitoring
├── save_full_snapshot.sh         # On-demand full conversation save
├── create_hourly_snapshots.sh    # Organize by clock hour
└── format_message_v2.jq          # Chat formatting logic
```

## Configuration

### Tracking Files

The system uses hidden files to track state:

```bash
/root/clawd/.last_save_line_count       # For token monitoring
/root/clawd/.last_snapshot_timestamp    # For incremental saves
/root/clawd/.token_warning_sent         # For warning deduplication
```

**Note:** Do not delete these files or incremental backups may duplicate content

### Session File Location

Default: `/root/.clawdbot/agents/main/sessions/*.jsonl`

If your session files are elsewhere, update the `SESSION_FILE` path in each script.

## Troubleshooting

### No snapshots being created

1. Check cron is running: `crontab -l`
2. Verify script has execute permission: `chmod +x scripts/*.sh`
3. Check logs: Run manually to see errors

### Messages breaking out of callouts

- Ensure `format_message_v2.jq` has the `gsub("\n\n"; "<br><br>")` line
- Check that all lines have `> ` prefix
- Verify jq is installed: `jq --version`

### Duplicated content in snapshots

- Delete tracking files and let system reset:
  ```bash
  rm /root/clawd/.last_snapshot_timestamp
  ```

### Empty callout boxes appearing

- Update `format_message_v2.jq` to filter empty messages
- Check for the `if ($text_content | length) > 0` condition

## Requirements

- **jq**: JSON parsing (`apt-get install jq`)
- **cron**: For automatic backups
- **Obsidian vault**: Target directory for markdown files

## Advanced Customization

### Change Backup Frequency

Edit crontab:
```bash
# Every 2 hours
0 */2 * * * /path/to/monitor_and_save.sh

# Every 30 minutes
*/30 * * * * /path/to/monitor_and_save.sh

# Specific times only (9am, 12pm, 6pm, 9pm)
0 9,12,18,21 * * * /path/to/monitor_and_save.sh
```

### Change Minimum Message Threshold

Edit `monitor_and_save.sh`:
```bash
# Change from 10 to 5 messages minimum
if [[ $new_lines -lt 5 ]]; then
```

### Add More Callout Styles

Obsidian callout types:
- `[!quote]` - Blue
- `[!check]` - Green
- `[!note]` - Cyan
- `[!tip]` - Purple
- `[!warning]` - Orange
- `[!danger]` - Red

### Customize Telegram Notifications

Edit `monitor_and_save.sh` to change warning text or add custom notifications.

## Best Practices

1. **Run hourly breakdown at end of day** - Use as organizational tool, not backup
2. **Keep incremental backups running** - This is your safety net
3. **Test scripts after setup** - Run manually first to verify output
4. **Backup tracking files** - Include `.last_snapshot_timestamp` in vault backups
5. **Use descriptive topic names** - For full snapshots, use meaningful names

## Example Workflow

**Daily routine:**
1. Automatic incremental backups run hourly (no action needed)
2. At end of day: `scripts/create_hourly_snapshots.sh 2026-01-20`
3. Review organized hourly files in Obsidian
4. Delete old incrementals if desired (hourly breakdown covers them)

**Before /new reset:**
1. Optional: `scripts/save_full_snapshot.sh before-reset`
2. Run `/new` safely - conversation is backed up
3. Continue chatting - incrementals resume automatically

## Integration with Clawdbot

This skill works with:
- **HEARTBEAT.md**: Automatic token monitoring
- **MEMORY.md**: Conversation archival system
- **Telegram integration**: Warning notifications
- **Any Obsidian vault**: Works with existing vaults

## Credits

Created by the Clawdbot community for reliable conversation backup and beautiful Obsidian formatting.