alerting

# alerting

Configure alert delivery for cron-monitor job failures and missed runs.

## When to use

- Setting up Slack notifications for job failures
- Adding webhook delivery to an incident management system
- Configuring email alerts for on-call rotation
- Testing that alert delivery is working before relying on it

## Alert Events

| Event | When it fires |
|---|---|
| `missed` | No ping received within the job's grace period after its scheduled time |
| `failure` | A ping was received with `status: "failure"` or a non-zero exit code |
| `recovery` | A success ping received after a `missed` or `failure` state |

## Create a Slack alert via API

```bash
curl -s -X POST https://monitor.example.com/api/alerts \
  -H "Authorization: Bearer cm_..." \
  -H "Content-Type: application/json" \
  -d '{
    "channel": "slack",
    "target": "https://hooks.slack.com/services/T0.../B0.../xxxx",
    "onMissed": 1,
    "onFailure": 1,
    "onRecovery": 1,
    "cooldownMinutes": 60
  }'
```

To apply to a specific job only, include `"jobId": "<uuid>"`. Omitting `jobId` applies to all jobs.

## Create a webhook alert via API

```bash
curl -s -X POST https://monitor.example.com/api/alerts \
  -H "Authorization: Bearer cm_..." \
  -H "Content-Type: application/json" \
  -d '{
    "channel": "webhook",
    "target": "https://api.pagerduty.com/v2/enqueue",
    "onMissed": 1,
    "onFailure": 1,
    "onRecovery": 0,
    "cooldownMinutes": 30,
    "jobId": "550e8400-e29b-41d4-a716-446655440000"
  }'
```

## Webhook payload format

POST to the target URL with JSON body and `X-CM-Signature` HMAC-SHA256 header:

```json
{
  "event": "failure",
  "job": {
    "id": "550e8400-...",
    "name": "API Sync",
    "slug": "api-sync",
    "schedule": "*/5 * * * *"
  },
  "execution": {
    "id": "7f8a9b2c-...",
    "status": "failure",
    "exitCode": 1,
    "durationMs": 410,
    "finishedAt": "2026-03-20T15:52:14Z",
    "output": "Connection timeout"
  },
  "timestamp": "2026-03-20T15:52:20Z",
  "monitorUrl": "https://monitor.example.com"
}
```

Verify the signature:

```javascript
const crypto = require('node:crypto');
const secret = 'cm_your_webhook_secret';
const signature = req.headers['x-cm-signature'];
const expected = crypto.createHmac('sha256', secret)
  .update(JSON.stringify(req.body))
  .digest('hex');
const valid = crypto.timingSafeEqual(
  Buffer.from(signature),
  Buffer.from(`sha256=${expected}`)
);
```

## Test an alert

```bash
curl -s -X POST https://monitor.example.com/api/alerts/<id>/test \
  -H "Authorization: Bearer cm_..."
# Response: { "ok": true, "channel": "slack", "deliveredAt": "2026-03-20T16:00:00Z" }
```

## List and manage alerts via API

```bash
# List all alert configs
curl -s https://monitor.example.com/api/alerts \
  -H "Authorization: Bearer cm_..."

# Update cooldown period
curl -s -X PATCH https://monitor.example.com/api/alerts/<id> \
  -H "Authorization: Bearer cm_..." \
  -H "Content-Type: application/json" \
  -d '{ "cooldownMinutes": 120 }'

# Delete
curl -s -X DELETE https://monitor.example.com/api/alerts/<id> \
  -H "Authorization: Bearer cm_..."
```

## Cooldown behavior

The `cooldownMinutes` setting prevents alert storms. Once an alert fires for a job event, the same alert type will not fire again for that job until the cooldown period passes. Recovery events always fire regardless of cooldown.

## Troubleshooting

### Alert not firing

1. Check the job status in the dashboard - alert only fires on status transitions
2. Verify the alert config applies to the correct job (or all jobs)
3. Check `onMissed`/`onFailure` flags are set to `1`
4. Check the cooldown: the same alert may be suppressed within the cooldown window

### Webhook getting 401/403

The receiving endpoint may require auth headers. Use a proxy or middleware that adds auth before forwarding to your upstream. cron-monitor does not support custom request headers on webhook delivery.