sensor-ingest

# sensor-ingest

Configure, test, and troubleshoot MQTT and HTTP POST ingest pipelines for weather-station-dashboard.

## When to Use

- Setting up a new sensor to send data via MQTT
- Configuring HTTP POST ingest from a Raspberry Pi or ESP32
- Debugging why a station is showing offline despite sending data
- Understanding which MQTT topics are accepted and which are rejected
- Testing the ingest pipeline end-to-end before deploying hardware

## Ingest Methods

### HTTP POST (always available)

Send readings to `POST /api/readings`. No broker required.

```bash
curl -X POST http://localhost:3005/api/readings \
  -H "Content-Type: application/json" \
  -d '{
    "station_id": "abc-123-uuid",
    "temperature_c": 22.4,
    "humidity_pct": 68.0,
    "pressure_hpa": 1013.2,
    "rainfall_mm": 0.0,
    "wind_speed_ms": 3.1,
    "wind_dir_deg": 225.0,
    "uv_index": 4,
    "soil_temp_c": 18.2,
    "soil_moisture_pct": 52.0
  }'
```

All sensor fields are optional except `station_id`. Send only the sensors your hardware supports.

### MQTT Ingest (optional, set MQTT_ENABLED=1)

Start the server with `MQTT_ENABLED=1` and set `MQTT_BROKER_URL`.

```bash
MQTT_ENABLED=1 \
MQTT_BROKER_URL=mqtt://192.168.1.100:1883 \
MQTT_TOPIC_PREFIX=farm/weather \
pnpm start
```

The server subscribes to `farm/weather/#` and matches incoming messages to stations by their configured `mqtt_topic` field.

## MQTT Topic Routing

Each station has an optional `mqtt_topic` field (set in Settings or via API). This is the full topic path the station publishes to.

Example station configuration:
```bash
# Update a station's MQTT topic
curl -X PUT http://localhost:3005/api/stations/{station_id} \
  -H "Content-Type: application/json" \
  -d '{ "mqtt_topic": "farm/weather/north-field" }'
```

When a message arrives on `farm/weather/north-field`, it is matched to the station with that mqtt_topic value and ingested. Messages on topics not matching any station are rejected and logged.

## MQTT Payload Format

Publish JSON to the station's topic. All numeric fields are optional - include only what your sensor reports:

```json
{
  "temperature_c": 22.4,
  "humidity_pct": 68.0,
  "pressure_hpa": 1013.2,
  "rainfall_mm": 0.0,
  "wind_speed_ms": 3.1,
  "wind_dir_deg": 225.0,
  "uv_index": 4,
  "soil_temp_c": 18.2,
  "soil_moisture_pct": 52.0
}
```

Do not include `station_id` in the MQTT payload - it is inferred from the topic.

## Testing MQTT Ingest

Use Mosquitto clients or any MQTT client:

```bash
# Publish a test reading
mosquitto_pub \
  -h 192.168.1.100 \
  -t "farm/weather/north-field" \
  -m '{"temperature_c":22.4,"humidity_pct":68,"pressure_hpa":1013.2}'

# Subscribe to verify the broker is receiving
mosquitto_sub -h 192.168.1.100 -t "farm/weather/#" -v
```

Then check if the reading appeared:
```bash
curl "http://localhost:3005/api/readings?station_id={id}&limit=5"
```

## Ingest Pipeline Steps

When a reading arrives (HTTP or MQTT):

1. Zod validation - reject if payload is malformed
2. Station lookup - reject if station_id not found or inactive
3. MQTT topic check - reject if topic does not match any station's mqtt_topic
4. INSERT into readings table with generated UUID and current timestamp
5. UPDATE station.last_seen to current time
6. Evaluate all enabled alert thresholds for this station
7. If threshold crossed: INSERT alert_event, broadcast WSMessage { type: 'alert' }
8. Broadcast WSMessage { type: 'reading' } to subscribed WebSocket clients
9. Return 201 Created (HTTP) or log success (MQTT)

## Troubleshooting

### Readings not appearing via HTTP POST

Check the response body for validation errors. Common issues:
- `station_id` is missing or not a valid UUID
- Station with that ID does not exist - create it first via `POST /api/stations`
- Station has `active: 0` - re-enable via `PUT /api/stations/{id}`

```bash
# Verify station exists
curl http://localhost:3005/api/stations/{station_id}
```

### MQTT messages not being ingested

1. Verify MQTT_ENABLED=1 in server environment
2. Check server logs for "MQTT connected" message on startup
3. Confirm the station's mqtt_topic matches the topic you are publishing to exactly
4. Check server logs for "REJECTED" entries - topic mismatch is the most common cause
5. Verify the JSON payload is valid: `echo '{"temperature_c":22}' | python3 -m json.tool`

### Station stays offline after sending readings

The station's `last_seen` field is updated on each successful ingest. If it is not updating:
- The reading may be failing validation silently - check server logs
- The station_id in HTTP POST or the mqtt_topic mapping may be wrong
- Confirm with `GET /api/readings?station_id={id}&limit=1` whether any readings exist

### MQTT reconnect behavior

The server uses the mqtt package with automatic reconnect (exponential backoff, max 30s). On reconnect it resubscribes to `MQTT_TOPIC_PREFIX/#`. No manual intervention is needed. Check `MQTT_BROKER_URL` is reachable from the server container/host.

### Data retention pruning

Raw readings older than `DATA_RETENTION_DAYS` (default 365) are deleted nightly at 00:05 by the same cron job that computes daily summaries. Daily summaries are not pruned. To change retention:

```bash
curl -X PUT http://localhost:3005/api/settings \
  -H "Content-Type: application/json" \
  -d '{ "data_retention_days": 180 }'
```