finetune-data-curator

# finetune-data-curator Skill

## When to Use

Use this skill when you need to:
- Create or curate JSONL datasets for LLM fine-tuning
- Validate dataset format compliance for OpenAI, Anthropic, or Llama training
- Find and remove near-duplicate samples from a dataset
- Score dataset quality and identify issues
- Split a dataset into train/eval subsets
- Convert datasets between OpenAI, Anthropic, and Llama formats
- Export cleaned datasets for training

## Quick Start

```bash
cd finetune-data-curator
cp .env.example .env
# edit .env: set ADMIN_KEY=<secure random string> and DATA_DIR=./data

pnpm install
pnpm dev   # server + SPA on :4400
```

Open http://localhost:4400.

## Docker Quick Start

```bash
ADMIN_KEY=mysecretkey docker compose up
# App: http://localhost:4400
```

## Creating a Dataset

Via the web UI: click "New Dataset", choose a format (OpenAI, Anthropic, or Llama), name it.

Via API:
```bash
curl -X POST http://localhost:4400/api/datasets \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-dataset", "format": "openai"}'
```

## Importing JSONL

Via the web UI: open a dataset, click Import, paste JSONL or upload a file.

Via API:
```bash
curl -X POST http://localhost:4400/api/datasets/ds_abc/samples \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"messages": [{"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi!"}]}]'
```

## Running Validation

Via the web UI: open a dataset, click "Run Validation".

Via API:
```bash
curl -X POST http://localhost:4400/api/datasets/ds_abc/validate \
  -H "X-Admin-Key: $ADMIN_KEY"
```

Validation checks: format compliance, missing fields, empty content, excessive length, degenerate samples.

## Finding Duplicates

```bash
curl -X POST http://localhost:4400/api/datasets/ds_abc/dedup \
  -H "X-Admin-Key: $ADMIN_KEY"
```

Returns pairs with Jaccard similarity above the configured threshold (default 0.8).

## Splitting

```bash
curl -X POST http://localhost:4400/api/datasets/ds_abc/split \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"ratio": 0.8, "seed": 42}'
```

Creates `ds_abc.train.jsonl` (80%) and `ds_abc.eval.jsonl` (20%).

## Exporting

```bash
# Download as OpenAI format
curl "http://localhost:4400/api/datasets/ds_abc/export?format=openai" \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -o my-dataset.jsonl

# Convert to Anthropic format on export
curl "http://localhost:4400/api/datasets/ds_abc/export?format=anthropic" \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -o my-dataset-anthropic.jsonl

# Export train split only
curl "http://localhost:4400/api/datasets/ds_abc/export?format=openai&split=train" \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -o train.jsonl
```

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `PORT` | `4400` | Server listen port |
| `DATA_DIR` | `./data` | Directory for JSONL files and index.json |
| `ADMIN_KEY` | (required) | Header value required for write operations |
| `MAX_UPLOAD_MB` | `50` | Maximum upload file size |