heartbeat-api

# heartbeat-api

The heartbeat pattern for monitoring scheduled jobs: jobs ping an API to prove they ran, and alerts fire if they don't.

## Pattern Overview

```
Backup Script                  Heartbeat Server
      |                               |
      |--- POST /ping/:id/start ----->|  (job began)
      |                               |
      |    [job runs]                 |
      |                               |
      |--- POST /ping/:id/complete -->|  (job finished)
      |                               |
      |                          [alert engine checks every minute]
      |                          [if not completed in window: fire alert]
```

## When to Use

- Any scheduled script that must run on time (backups, ETL, reports, cleanups)
- Cron jobs where you need to know if they silently failed
- Long-running jobs where you need duration tracking

## Key Design Decisions

### Why heartbeat instead of log scraping

Log scraping is fragile. The heartbeat pattern is explicit: the job actively asserts it ran and succeeded. No false negatives from log format changes.

### Window calculation

```
alert_time = last_scheduled_run + expected_duration + grace_period
```

Example: job scheduled at 02:00, expected 5 minutes, grace 30 minutes.
Alert fires at 02:35 if no completion ping received.

### Deduplication

Store the window start time in each alert. Before sending, check:
```sql
SELECT id FROM alerts
WHERE job_id = ? AND window_start = ? AND type = 'missed'
LIMIT 1
```

If found, skip. Prevents alert spam on restart or missed cleanup.

## Express Route Pattern

```typescript
// POST /ping/:jobId/start
router.post('/:jobId/start', validateApiKey, async (req, res) => {
  const job = await db.getJob(req.params.jobId);
  if (!job) return res.status(404).json({ error: 'Job not found' });

  const execution = await db.createExecution({
    jobId: job.id,
    startedAt: new Date().toISOString(),
    status: 'running',
  });

  await db.updateJobStatus(job.id, 'running');

  return res.json({ ok: true, executionId: execution.id });
});

// POST /ping/:jobId/complete
router.post('/:jobId/complete', validateApiKey, async (req, res) => {
  const { executionId, exitCode } = req.body;
  const execution = await db.getExecution(executionId);

  const durationS = Math.round(
    (Date.now() - new Date(execution.startedAt).getTime()) / 1000
  );

  await db.updateExecution(executionId, {
    completedAt: new Date().toISOString(),
    status: 'completed',
    durationS,
    exitCode: exitCode ?? 0,
  });

  await db.updateJobStatus(execution.jobId, 'ok');

  return res.json({ ok: true, duration_s: durationS });
});
```

## Alert Engine Pattern

```typescript
// Runs every minute via node-cron
async function checkMissedJobs(): Promise<void> {
  const jobs = await db.getActiveJobs();
  const now = new Date();

  for (const job of jobs) {
    if (job.status === 'paused') continue;

    const lastWindow = getLastScheduledRun(job.schedule, now);
    const alertTime = addMinutes(lastWindow, job.expectedDurM + job.gracePeriodM);

    if (now < alertTime) continue; // not yet overdue
    if (job.status === 'completed' || job.status === 'running') continue;

    // Check deduplication
    const alreadyAlerted = await db.hasAlert(job.id, lastWindow);
    if (alreadyAlerted) continue;

    await dispatchAlert(job, lastWindow, 'missed');
  }
}
```

## Cron Expression Parsing

Use `cron-parser` to find the last scheduled run:

```typescript
import { parseExpression } from 'cron-parser';

function getLastScheduledRun(cronExpr: string, now: Date): Date {
  const interval = parseExpression(cronExpr, { currentDate: now });
  return interval.prev().toDate();
}
```

## Webhook Payload Format

```json
{
  "event": "missed",
  "job": {
    "id": "job_a1b2c3",
    "name": "nightly-postgres",
    "schedule": "0 2 * * *"
  },
  "window": {
    "scheduledAt": "2026-03-20T02:00:00Z",
    "alertFiredAt": "2026-03-20T02:35:00Z",
    "gracePeriodMinutes": 30
  },
  "dashboardUrl": "http://localhost:4000/jobs/job_a1b2c3"
}
```

## Email Template

Subject: `[backup-monitor] MISSED: nightly-postgres`

Body: Plain text. Include job name, schedule, last successful run, dashboard link.

## Dependencies

| Package | Purpose |
|---|---|
| `cron-parser` | Parse cron expressions, find last/next run |
| `node-cron` | Run alert engine on schedule |
| `nodemailer` | Send email alerts |
| `better-sqlite3` | SQLite storage |