backup-monitor

# backup-monitor

Monitor backup jobs with a heartbeat API. Get alerted when backups miss their window.

## When to Use

- Adding reliability monitoring to existing backup scripts
- Getting email/webhook alerts when a backup job does not run
- Tracking backup execution history and duration trends
- Verifying that backups completed before a restore operation

## Prerequisites

1. backup-monitor server running (self-hosted or Docker)
2. Backup scripts instrumented with curl ping calls
3. SMTP or webhook configured for alerts

## Installation (Docker)

```bash
docker compose up -d
# Dashboard at http://localhost:4000
```

## Quick Start

### Step 1 - Register a job

```bash
backup-monitor job create \
  --name nightly-postgres \
  --schedule "0 2 * * *" \
  --duration 5 \
  --grace 30 \
  --email ops@example.com
```

Output:
```
ok  job registered
--  id       job_a1b2c3d4e5f6
--  api key  bkm_xK9m4Pp8rQjL2TvN6yBd...

    Save this API key now. It will not be shown again.
```

### Step 2 - Instrument your backup script

```bash
#!/bin/bash
JOB_ID="job_a1b2c3d4e5f6"
API_KEY="bkm_xK9m4Pp8rQjL2TvN6yBd..."
BASE_URL="http://localhost:4000"

EXEC_ID=$(curl -s -X POST "$BASE_URL/ping/$JOB_ID/start" \
  -H "Authorization: Bearer $API_KEY" | jq -r '.executionId')

# Your backup command here
pg_dump mydb > /backups/mydb-$(date +%Y%m%d).sql
EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  curl -s -X POST "$BASE_URL/ping/$JOB_ID/complete" \
    -H "Authorization: Bearer $API_KEY" \
    -d "{\"executionId\":\"$EXEC_ID\",\"exitCode\":0}"
else
  curl -s -X POST "$BASE_URL/ping/$JOB_ID/fail" \
    -H "Authorization: Bearer $API_KEY" \
    -d "{\"executionId\":\"$EXEC_ID\",\"exitCode\":$EXIT_CODE}"
fi
```

## CLI Reference

| Command | Description |
|---|---|
| `backup-monitor job create [flags]` | Register a new backup job |
| `backup-monitor job list` | List all jobs with status |
| `backup-monitor job delete <id>` | Delete a job |
| `backup-monitor ping start <id>` | Send start ping |
| `backup-monitor ping complete <id>` | Send completion ping |
| `backup-monitor ping fail <id>` | Send failure ping |
| `backup-monitor --help` | Show help |

## Job Create Flags

| Flag | Description | Default |
|---|---|---|
| `--name <name>` | Job identifier (required) | - |
| `--schedule <cron>` | Cron expression (required) | - |
| `--duration <min>` | Expected duration in minutes | 60 |
| `--grace <min>` | Grace period before alert fires | 30 |
| `--email <addr>` | Alert email address | global setting |
| `--webhook <url>` | Alert webhook URL | global setting |
| `--retention <days>` | History retention | 90 |

## Heartbeat API

All endpoints require `Authorization: Bearer <api-key>`.

```
POST /ping/:jobId/start       # Job started
POST /ping/:jobId/complete    # Job completed successfully
POST /ping/:jobId/fail        # Job failed
GET  /ping/:jobId/status      # Current status (no auth required)
```

## Dashboard API

```
GET  /api/jobs                # All jobs with status
POST /api/jobs                # Register job
GET  /api/jobs/:id/history    # Execution history
GET  /api/alerts              # Alert log
PATCH /api/settings           # Update SMTP/webhook settings
```

## Environment Variables

| Variable | Description | Default |
|---|---|---|
| `PORT` | Server port | 4000 |
| `NODE_ENV` | development or production | development |
| `DATABASE_PATH` | SQLite file path | ./data/backup-monitor.db |
| `DASHBOARD_USER` | Basic auth username | (disabled) |
| `DASHBOARD_PASS` | Basic auth password | (disabled) |
| `SMTP_HOST` | SMTP server | - |
| `SMTP_PORT` | SMTP port | 587 |
| `SMTP_USER` | SMTP username | - |
| `SMTP_PASS` | SMTP password | - |
| `SMTP_FROM` | From address | backup-monitor@localhost |
| `ALERT_CHECK_INTERVAL` | Alert check cron | * * * * * |
| `LOG_LEVEL` | debug, info, warn, error | info |

## Alert Behavior

- Alerts fire when: job missed its window (started time + expected duration + grace period)
- Alerts fire when: job sent a fail ping
- Recovery alert fires when: a previously missed/failed job completes successfully
- Duplicate alerts suppressed: same window = same alert (no spam)
- Job pause: suppress alerts without disabling monitoring

## Troubleshooting

### No alert received for missed backup

Check:
1. Grace period has passed (default 30 minutes after expected completion)
2. SMTP/webhook configured in Settings
3. Alert log in dashboard for delivery errors
4. Server logs: `docker logs backup-monitor`

### "Unauthorized" on ping

API key does not match. Each job has its own key shown once on creation. To reset, delete and recreate the job.

### Job showing "running" but backup is done

The completion ping was not sent (check exit code handling in your script) or the ping failed (check network connectivity to the monitor server).