cron-scheduling

# Cron & Scheduling

Schedule and manage recurring tasks. Covers cron syntax, crontab management, systemd timers, one-off scheduling, timezone handling, monitoring, and common failure patterns.

## When to Use

- Running scripts on a schedule (backups, reports, cleanup)
- Setting up systemd timers (modern cron alternative)
- Debugging why a scheduled job didn't run
- Handling timezones in scheduled tasks
- Monitoring and alerting on job failures
- Running one-off delayed commands

## Cron Syntax

### The five fields

```
┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12 or JAN-DEC)
│ │ │ │ ┌─ day of week (0-7, 0 and 7 = Sunday, or SUN-SAT)
│ │ │ │ │
* * * * * command
```

### Common schedules

```bash
# Every minute
* * * * * /path/to/script.sh

# Every 5 minutes
*/5 * * * * /path/to/script.sh

# Every hour at :00
0 * * * * /path/to/script.sh

# Every day at 2:30 AM
30 2 * * * /path/to/script.sh

# Every Monday at 9:00 AM
0 9 * * 1 /path/to/script.sh

# Every weekday at 8:00 AM
0 8 * * 1-5 /path/to/script.sh

# First day of every month at midnight
0 0 1 * * /path/to/script.sh

# Every 15 minutes during business hours (Mon-Fri 9-17)
*/15 9-17 * * 1-5 /path/to/script.sh

# Twice a day (9 AM and 5 PM)
0 9,17 * * * /path/to/script.sh

# Every quarter (Jan, Apr, Jul, Oct) on the 1st at midnight
0 0 1 1,4,7,10 * /path/to/script.sh

# Every Sunday at 3 AM
0 3 * * 0 /path/to/script.sh
```

### Special strings (shorthand)

```bash
@reboot    /path/to/script.sh   # Run once at startup
@yearly    /path/to/script.sh   # 0 0 1 1 *
@monthly   /path/to/script.sh   # 0 0 1 * *
@weekly    /path/to/script.sh   # 0 0 * * 0
@daily     /path/to/script.sh   # 0 0 * * *
@hourly    /path/to/script.sh   # 0 * * * *
```

## Crontab Management

```bash
# Edit current user's crontab
crontab -e

# List current crontab
crontab -l

# Edit another user's crontab (root)
sudo crontab -u www-data -e

# Remove all cron jobs (be careful!)
crontab -r

# Install crontab from file
crontab mycrontab.txt

# Backup crontab
crontab -l > crontab-backup-$(date +%Y%m%d).txt
```

### Crontab best practices

```bash
# Set PATH explicitly (cron has minimal PATH)
PATH=/usr/local/bin:/usr/bin:/bin

# Set MAILTO for error notifications
MAILTO=admin@example.com

# Set shell explicitly
SHELL=/bin/bash

# Full crontab example
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=admin@example.com
SHELL=/bin/bash

# Backups
0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

# Cleanup old logs
0 3 * * 0 find /var/log/myapp -name "*.log" -mtime +30 -delete

# Health check
*/5 * * * * /opt/scripts/healthcheck.sh || /opt/scripts/alert.sh "Health check failed"
```

## Systemd Timers

### Create a timer (modern cron replacement)

```ini
# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=backup
StandardOutput=journal
StandardError=journal
```

```ini
# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup daily at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target
```

```bash
# Enable and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

# Check timer status
systemctl list-timers
systemctl list-timers --all

# Check last run
systemctl status backup.service
journalctl -u backup.service --since today

# Run manually (for testing)
sudo systemctl start backup.service

# Disable timer
sudo systemctl disable --now backup.timer
```

### OnCalendar syntax

```ini
# Systemd calendar expressions

# Daily at midnight
OnCalendar=daily
# or: OnCalendar=*-*-* 00:00:00

# Every Monday at 9 AM
OnCalendar=Mon *-*-* 09:00:00

# Every 15 minutes
OnCalendar=*:0/15

# Weekdays at 8 AM
OnCalendar=Mon..Fri *-*-* 08:00:00

# First of every month
OnCalendar=*-*-01 00:00:00

# Every 6 hours
OnCalendar=0/6:00:00

# Specific dates
OnCalendar=2026-02-03 12:00:00

# Test calendar expressions
systemd-analyze calendar "Mon *-*-* 09:00:00"
systemd-analyze calendar "*:0/15"
systemd-analyze calendar --iterations=5 "Mon..Fri *-*-* 08:00:00"
```

### Advantages over cron

```
Systemd timers vs cron:
+ Logs in journald (journalctl -u service-name)
+ Persistent: catches up on missed runs after reboot
+ RandomizedDelaySec: prevents thundering herd
+ Dependencies: can depend on network, mounts, etc.
+ Resource limits: CPUQuota, MemoryMax, etc.
+ No lost-email problem (MAILTO often misconfigured)
- More files to create (service + timer)
- More verbose configuration
```

## One-Off Scheduling

### at (run once at a specific time)

```bash
# Schedule a command
echo "/opt/scripts/deploy.sh" | at 2:00 AM tomorrow
echo "reboot" | at now + 30 minutes
echo "/opt/scripts/report.sh" | at 5:00 PM Friday

# Interactive (type commands, Ctrl+D to finish)
at 10:00 AM
> /opt/scripts/task.sh
> echo "Done" | mail -s "Task complete" admin@example.com
> <Ctrl+D>

# List pending jobs
atq

# View job details
at -c <job-number>

# Remove a job
atrm <job-number>
```

### sleep-based (simplest)

```bash
# Run something after a delay
(sleep 3600 && /opt/scripts/task.sh) &

# With nohup (survives logout)
nohup bash -c "sleep 7200 && /opt/scripts/task.sh" &
```

## Timezone Handling

```bash
# Cron runs in the system timezone by default
# Check system timezone
timedatectl
date +%Z

# Set timezone for a specific cron job
# Method 1: TZ variable in crontab
TZ=America/New_York
0 9 * * * /opt/scripts/report.sh

# Method 2: In the script itself
#!/bin/bash
export TZ=UTC
# All date operations now use UTC

# Method 3: Wrapper
TZ=Europe/London date '+%Y-%m-%d %H:%M:%S'

# List available timezones
timedatectl list-timezones
timedatectl list-timezones | grep America
```

### DST pitfalls

```
Problem: A job scheduled for 2:30 AM may run twice or not at all
during DST transitions.

"Spring forward": 2:30 AM doesn't exist (clock jumps 2:00 → 3:00)
"Fall back": 2:30 AM happens twice

Mitigation:
1. Schedule critical jobs outside 1:00-3:00 AM
2. Use UTC for the schedule: TZ=UTC in crontab
3. Make jobs idempotent (safe to run twice)
4. Systemd timers handle DST correctly
```

## Monitoring and Debugging

### Why didn't my cron job run?

```bash
# 1. Check cron daemon is running
systemctl status cron    # Debian/Ubuntu
systemctl status crond   # CentOS/RHEL

# 2. Check cron logs
grep CRON /var/log/syslog           # Debian/Ubuntu
grep CRON /var/log/cron             # CentOS/RHEL
journalctl -u cron --since today    # systemd

# 3. Check crontab actually exists
crontab -l

# 4. Test the command manually (with cron's environment)
env -i HOME=$HOME SHELL=/bin/sh PATH=/usr/bin:/bin /opt/scripts/backup.sh
# If it fails here but works normally → PATH or env issue

# 5. Check permissions
ls -la /opt/scripts/backup.sh   # Must be executable
ls -la /var/spool/cron/         # Crontab file permissions

# 6. Check for syntax errors in crontab
# cron silently ignores lines with errors

# 7. Check if output is being discarded
# By default, cron emails output. If no MTA, output is lost.
# Always redirect: >> /var/log/myjob.log 2>&1
```

### Job wrapper with logging and alerting

```bash
#!/bin/bash
# cron-wrapper.sh — Run a command with logging, timing, and error alerting
# Usage: cron-wrapper.sh <job-name> <command> [args...]

set -euo pipefail

JOB_NAME="${1:?Usage: cron-wrapper.sh <job-name> <command> [args...]}"
shift
COMMAND=("$@")

LOG_DIR="/var/log/cron-jobs"
mkdir -p "$LOG_DIR"
LOG_FILE="$LOG_DIR/$JOB_NAME.log"

log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" >> "$LOG_FILE"; }

log "START: ${COMMAND[*]}"
START_TIME=$(date +%s)

if "${COMMAND[@]}" >> "$LOG_FILE" 2>&1; then
    ELAPSED=$(( $(date +%s) - START_TIME ))
    log "SUCCESS (${ELAPSED}s)"
else
    EXIT_CODE=$?
    ELAPSED=$(( $(date +%s) - START_TIME ))
    log "FAILED with exit code $EXIT_CODE (${ELAPSED}s)"
    # Alert (customize as needed)
    echo "Cron job '$JOB_NAME' failed with exit $EXIT_CODE" | \
        mail -s "CRON FAIL: $JOB_NAME" admin@example.com 2>/dev/null || true
    exit $EXIT_CODE
fi
```

```bash
# Use in crontab:
0 2 * * * /opt/scripts/cron-wrapper.sh daily-backup /opt/scripts/backup.sh
*/5 * * * * /opt/scripts/cron-wrapper.sh health-check /opt/scripts/healthcheck.sh
```

### Lock to prevent overlap

```bash
# Prevent concurrent runs (job takes longer than interval)
# Method 1: flock
* * * * * flock -n /tmp/myjob.lock /opt/scripts/slow-job.sh

# Method 2: In the script
LOCKFILE="/tmp/myjob.lock"
exec 200>"$LOCKFILE"
flock -n 200 || { echo "Already running"; exit 0; }
# ... do work ...
```

## Idempotent Job Patterns

```bash
# Idempotent backup (only creates if newer than last backup)
#!/bin/bash
BACKUP_DIR="/backups/$(date +%Y%m%d)"
[[ -d "$BACKUP_DIR" ]] && { echo "Backup already exists"; exit 0; }
mkdir -p "$BACKUP_DIR"
pg_dump mydb > "$BACKUP_DIR/mydb.sql"

# Idempotent cleanup (safe to run multiple times)
find /tmp/uploads -mtime +7 -type f -delete 2>/dev/null || true

# Idempotent sync (rsync only transfers changes)
rsync -az /data/ backup-server:/backups/data/
```

## Tips

- Always redirect output in cron jobs: `>> /var/log/job.log 2>&1`. Without this, output goes to mail (if configured) or is silently lost.
- Test cron jobs by running them with `env -i` to simulate cron's minimal environment. Most failures are caused by missing `PATH` or environment variables.
- Use `flock` to prevent overlapping runs when a job might take longer than its schedule interval.
- Make all scheduled jobs idempotent. If a job runs twice (DST, manual trigger, crash recovery), it should produce the same result.
- `systemd-analyze calendar` is invaluable for verifying timer schedules before deploying.
- Never schedule critical jobs between 1:00 AM and 3:00 AM if DST applies. Use UTC schedules instead.
- Log the start time, end time, and exit code of every cron job. Without this, debugging failures after the fact is guesswork.
- Prefer systemd timers over cron for production services: you get journald logging, missed-run catchup (`Persistent=true`), and resource limits for free.