Skill: JSON log parsing

# Skill: JSON log parsing

## When to use

Use this skill when implementing the log parser, handling structured log output from Node.js services using bunyan or pino, mapping numeric log levels to named levels, extracting fields from JSON log lines, or falling back to plain-text parsing when JSON is absent.

## Log format auto-detection

The parser applies the following algorithm to each incoming line:

1. Attempt `JSON.parse(line)`
2. If the parse succeeds, treat as structured JSON log
3. Extract `level`, `time`, `msg` (or `message`) fields from the parsed object
4. If the parse fails, treat the entire line as a plain-text message with level `info` and timestamp from wall clock

This matches the behavior of bunyan, pino, and Winston JSON transports.

## JSON log structure

A minimal valid structured log line:

```json
{"level":30,"time":1742040276003,"msg":"server started","pid":1,"hostname":"api"}
```

Field mapping:

| JSON field | Type | Description |
|---|---|---|
| `level` | number or string | Log level. Numbers follow bunyan convention. |
| `time` | number | Unix millisecond timestamp |
| `msg` or `message` | string | Human-readable message |
| All other fields | any | Stored as structured fields, shown in expanded view |

## Bunyan numeric level mapping

| Number | Named level | Color |
|---|---|---|
| 10 | trace | gray |
| 20 | debug | gray |
| 30 | info | white |
| 40 | warn | yellow |
| 50 | error | red |
| 60 | fatal | red (bright) |

Numbers outside these values are rounded down to the nearest tier. Numbers below 10 are treated as trace. Numbers above 60 are treated as fatal.

## Pino level strings

Pino can be configured to emit string levels. The parser accepts both numeric and string level values:

```json
{"level":"warn","time":1742040276003,"msg":"high memory usage"}
```

String levels accepted (case-insensitive): `trace`, `debug`, `info`, `warn`, `warning`, `error`, `fatal`, `critical`.

`warning` maps to `warn`. `critical` maps to `fatal`.

## Implementation reference

### LogLine interface

```typescript
interface LogLine {
  id:        string;         // UUID, assigned by parser
  source_id: string;         // source UUID
  ts:        number;         // Unix ms
  level:     LogLevel;       // 'trace' | 'debug' | 'info' | 'warn' | 'error' | 'fatal'
  message:   string;
  fields:    Record<string, unknown> | null;
  raw:       string;         // original line text
}
```

### Parser function signature

```typescript
function parseLine(raw: string, sourceId: string): LogLine
```

### Level detection

```typescript
function detectLevel(parsed: Record<string, unknown>): LogLevel {
  const v = parsed['level'];
  if (typeof v === 'number') return numericToLevel(v);
  if (typeof v === 'string') return stringToLevel(v);
  return 'info';
}

function numericToLevel(n: number): LogLevel {
  if (n >= 60) return 'fatal';
  if (n >= 50) return 'error';
  if (n >= 40) return 'warn';
  if (n >= 30) return 'info';
  if (n >= 20) return 'debug';
  return 'trace';
}
```

### Timestamp extraction

```typescript
function extractTimestamp(parsed: Record<string, unknown>): number {
  const t = parsed['time'];
  if (typeof t === 'number' && t > 0) return t;
  // pino uses epoch ms; bunyan uses epoch ms
  // fall back to wall clock
  return Date.now();
}
```

### Field extraction

After extracting `level`, `time`, and `msg`/`message`, the remaining keys in the parsed object are stored as `fields`. Reserved keys to exclude from fields:

- `level`
- `time`
- `msg`
- `message`
- `v` (bunyan schema version)
- `pid`
- `hostname`
- `name`

All remaining top-level keys are stored in the `fields` map.

### Plain text fallback

When JSON parse fails, construct a LogLine with:

- `level`: `info`
- `ts`: `Date.now()`
- `message`: the full raw line with ANSI escape codes stripped
- `fields`: `null`
- `raw`: the original unmodified line

Strip ANSI codes using: `/\x1b\[[0-9;]*m/g`

## Filter engine

The filter engine operates on a `FilterState`:

```typescript
interface FilterState {
  minLevel:  LogLevel;           // inclusive lower bound
  sources:   string[] | null;    // null = all sources
  search:    string | null;      // regex pattern or null
}
```

A `LogLine` passes the filter when:

1. `levelOrder[line.level] >= levelOrder[filter.minLevel]`
2. `filter.sources === null || filter.sources.includes(line.source_id)`
3. `filter.search === null || new RegExp(filter.search, 'i').test(line.message)`

Level order:

```typescript
const levelOrder: Record<LogLevel, number> = {
  trace: 0, debug: 1, info: 2, warn: 3, error: 4, fatal: 5
};
```

Precompile the `RegExp` once per filter state change, not per line.

## SSE stream format

Log lines are pushed to connected clients over Server-Sent Events. Each event is a JSON-encoded `LogLine`:

```
data: {"id":"...","source_id":"...","ts":1742040276003,"level":"error","message":"...","fields":{...},"raw":"..."}\n\n
```

The SSE endpoint is `GET /api/stream`. Clients reconnect automatically using the browser's built-in `EventSource` reconnect behavior. The server sends a `data: ping\n\n` heartbeat every 15 seconds to keep the connection alive through proxies.

On initial connect, the server replays the ring buffer for each active source filtered by the request's `?level=` and `?source=` query parameters. Replay events are sent as type `replay` before switching to live `data` events:

```
event: replay
data: {...}\n\n
```

## Common pitfalls

### Treating the level field as always a number

Pino allows string levels. The parser must handle both `30` and `"info"` for the same level. Always check `typeof v`.

### Crashing on malformed JSON

Not every line from a process is valid JSON. A syntax error in the log line must not crash the parser. Wrap `JSON.parse` in a try/catch and fall back to plain text.

### Using `eval` or `Function` constructor for parsing

Use only `JSON.parse`. Never eval user-supplied log content.

### Storing `raw` only as parsed

Always store the original `raw` string unmodified. The dashboard may need to display the original line for plain-text sources or for debugging the parser.

### Off-by-one in level thresholds

The `minLevel` filter is inclusive. A filter of `warn` should show `warn`, `error`, and `fatal` lines. Verify with: `levelOrder[line.level] >= levelOrder[filter.minLevel]`.