explore-dnn-model

# Explore DNN Model

## Minimum Required Inputs (Hard Requirement)

To use this skill, the user must provide:
- A model checkpoint / model file(s) as a **local** file or directory path (it may be outside the workspace).

If the user provides only the checkpoint path (no model name, repo link, or source code), proceed by:
1) Attempting to identify the model name/family from the checkpoint file/dir itself (filenames, adjacent configs/README, embedded metadata, `state_dict` key patterns, etc.).
2) Searching for the implementation in the workspace and/or alongside the checkpoint directory (e.g., nearby Python packages, inference scripts, config files).
3) If still not found, using the best-guess model name/family to search online for the canonical implementation, then cloning the upstream source into `tmp/<experiment-dir>/refs/` for investigation (prefer shallow clone; record URL + commit/tag used).

## Goals

This skill has three goals:

1) Verify that the given DNN model can work (inference or training; default focus is **inference**) in the *current* Python environment of the workspace.
2) Determine how to use it (inference or training; default is **inference**) by reading the upstream source code and producing minimal, reproducible runs.
3) Produce two reports:
   - **Experiment report** (programmatic): generated from `tmp/<experiment-dir>/outputs/` with minimal/no reasoning.
   - **Stakeholder report** (agent-written): generated by the agent from the experiment report + outputs/logs, with deeper analysis and recommendations.

The reports cover:
   - Input and output contracts (formats, shapes, dtypes, preprocessing/postprocessing)
   - Benchmarks and performance profiling (latency/throughput/memory, device details)
   - User-provided metrics/targets (e.g., accuracy, mAP, IoU, F1, latency budget), and whether/how they are met

Before changing anything, detect how the environment is managed by checking for:
- `pixi.toml` and/or `pyproject.toml` (Pixi-managed project)
- `.venv/` (venv-managed project)

## Dependency Policy (Ask Once, Then Apply)

If any dependency is missing:
- Do **not** install it automatically *without user confirmation*.
- List the missing packages (and versions/constraints if known) and ask the developer how to proceed.
- Provide clear options, let the developer choose, then proceed with the chosen approach.
- Once the developer confirms an approach, apply it for **all** newly required packages (no need to ask approval per package).

### Version Strategy

- First attempt: use the **latest versions** resolved by the selected package manager (`pixi`, `pip`, `uv`).
- If that fails (import/runtime errors, incompatibilities): fall back to the **specific versions/constraints** documented by the model’s upstream source code or docs.

### Preferred Options (in order)

**Pixi-managed env**
- Ask the user to choose one:
  - Modify the current Pixi environment by adding deps to the relevant manifest (`pixi.toml` / `pyproject.toml`).
  - Create a new Pixi environment specifically to test this model.
- Then use `pixi install`/`pixi run ...` to execute.
- Prefer **PyPI** packages over **conda-forge** when both are available.
- Avoid direct `pip install ...` into the Pixi environment unless the developer explicitly requests it.

**`.venv`-managed env**
Ask the user to choose one:
- Install deps via `pip` (or `uv pip`) into the current `.venv`.
- Create a new venv specifically for this model (keeps the repo venv clean).

## Inputs to Collect (ask if missing)

- Model name and/or upstream repo link and/or source code path (optional but speeds up identification)
- Model task/modality if unclear (classification/detection/segmentation/embedding/audio/video/etc.)
- Checkpoint path (file/dir) and format (`.pt`, `.pth`, `.onnx`, `.engine`, etc.)
- Any known I/O contract details (expected resolution, channel order, normalization, label mapping), if the user has them
- CPU-only requirement (only if the user explicitly requests CPU-only)
- Optional: user-provided metrics/targets to evaluate (quality and/or performance)

Notes:
- Determine framework/runtime automatically from checkpoint type + upstream code/docs + what’s available in the current Python environment.
- If hardware is unspecified, default to using hardware acceleration when available (CUDA GPU, ROCm GPU, Apple MPS, etc.). Use CPU-only only if the user requested it.
- If unspecified, the default objective is to confirm the model runs end-to-end from input → output (prefer real inputs found in the workspace; synthesize as a fallback) and record end-to-end timing.

## Core Workflow

### 0) Confirm artifacts and pick the target environment

- Confirm the minimum required inputs are present:
  - Checkpoint/model path is accessible locally (file/dir exists). It may be outside the workspace.
  - If model name/repo/source path is not provided, start by inferring it from the checkpoint and nearby files; if needed, locate it online and clone into `tmp/<experiment-dir>/refs/`.
- Detect environment type:
  - If both Pixi and `.venv` exist, ask the user which one should be treated as the “current” environment for this exploration.
- Device default:
  - If the user did not request CPU-only, use hardware acceleration when available (CUDA/ROCm/MPS/etc.).

### 1) Locate and read the upstream source code/docs

- First try to find the implementation locally:
  - Search the workspace and the checkpoint directory for source code, inference scripts, configs, and docs.
  - Prefer local source if it appears to be the canonical/official implementation for the checkpoint.
- If local source is not available or is clearly incomplete, use online search to find the canonical implementation:
  - Official GitHub repo, paper, model card, or vendor docs.
  - Check out the upstream repo under `tmp/<experiment-dir>/refs/<repo-name>` using a shallow clone (`--depth=1`), pinning a tag/commit when possible.
- Download/check out the relevant source code (pin a tag/commit when possible) and identify:
  - The exact inference entrypoints (scripts/modules), model class, preprocessing, postprocessing, and label mapping.
  - Any config files required to construct the model (YAML/JSON/TOML).
- Do not “guess” preprocessing/postprocessing: confirm from code and/or reference examples.

### 2) Derive required dependencies

Before running the model or changing the environment, determine the minimal dependencies required to run the model by using (in priority order):
- Upstream source code (setup files, `requirements*.txt`, `pyproject.toml`, import graph).
- Upstream docs/model card (pinned versions, known-good combos).
- Checkpoint type (e.g., `.onnx` implies ONNX Runtime; `.pt/.pth` implies PyTorch; `.engine` implies TensorRT).

Make a concise dependency list covering:
- Runtime/framework (e.g., `torch`, `onnxruntime`, `opencv-python`)
- Model-specific libs (e.g., `ultralytics`, `timm`, `transformers`, `mmengine`, etc.)
- Utility deps used by the official inference path (e.g., `numpy`, `Pillow`, `pyyaml`)
- Optional acceleration deps (CUDA/TensorRT) separated from the CPU baseline

### 3) Resolve missing dependencies (with user choice)

- Check whether each required dependency is available in the current environment.
- If anything is missing, ask the user which path to take:
  - **Pixi:** modify current manifest to add deps, or create a new Pixi env for this model.
  - **Venv:** install into current `.venv`, or create a new venv for this model.
- After the user confirms, apply the decision for all required packages (no per-package prompts).
- Use the **Version Strategy** above (latest first; fall back to pinned versions if needed).
- After dependency changes, run a quick smoke test:
  - Imports for the core runtime stack
  - Minimal “load model” path (without a full benchmark yet)

### 4) Ensure the checkpoint exists locally

- Do **not** download checkpoints automatically.
- Developers must provide checkpoints/model files (local file/dir paths).
- If the checkpoint is missing or only a URL is provided, ask the developer to download it and provide the local path.
- If the developer wants a conventional location, prefer `checkpoints/` (gitignored).
- Record provenance in a short note (based on what the developer provides):
  - Claimed source URL(s) or repo, version/commit/tag (if known), file size, and (if feasible) SHA256.

### 5) Create an experiment workspace under `tmp/`

Default experiment directory:

`<workspace>/tmp/<experiment-slug>-<time>`

If the user specifies a different location/name, use the user-provided one instead.

Create the standard directory layout:

```
tmp/<experiment-dir>/
  README.md     # experiment intent + directory guide (keep updated)
  refs/         # checked-out upstream repos (use shallow clone for online checkouts)
    README.md
  scripts/      # throwaway but reproducible scripts (committed if useful)
    README.md
  inputs/       # downloaded/synthesized test inputs
    README.md
  outputs/      # artifacts + machine-readable stats (e.g., `stats.json`)
    README.md
  logs/         # logs (stdout/stderr, profiling traces, command transcripts)
    README.md
  reports/      # markdown notes: what was tried, params, results
    README.md
    figures/    # images embedded in reports
    experiment-report.md
    stakeholder-report.md
```

Shell safety note (avoid accidental directory names):
- Do **not** use bash brace expansion to create these folders (e.g., `mkdir -p "$exp"/{refs,scripts,...}`), because quoting/spacing mistakes can create literal directories like `{refs,scripts,...}`.
- Prefer a simple loop or explicit `mkdir -p` calls, for example:

  ```
  exp="tmp/<experiment-dir>"
  mkdir -p "$exp"
  for d in refs scripts inputs outputs logs reports reports/figures; do
    mkdir -p "$exp/$d"
  done
  ```

Conventions:
- Use relative paths from `tmp/<experiment-dir>` in scripts so the folder is movable.
- Keep scripts small and single-purpose (`01_download_inputs.py`, `10_infer.py`, `20_visualize.py`, …).
- Run Python via the selected environment manager:
  - Pixi: `pixi run python ...`
  - Venv: use the venv’s Python (avoid system Python)

README requirements:
- Create `tmp/<experiment-dir>/README.md` to describe:
  - The intention of the experiment (what model, what checkpoint, what question you’re answering)
  - How to reproduce (one-line pointer to the primary script(s))
  - A brief map of what each top-level subdir contains
- Each top-level subdir must have its own `README.md` that:
  - Describes what belongs in the folder
  - Notes any important changes (append a short “Changes” section as you iterate)

### 6) Collect or synthesize inputs

- First try to find suitable inputs already present in the workspace (e.g., under `datasets/`, `downloads/`, or other project-specific data dirs) based on what you learned from the checkpoint/source code (task, modality, expected resolution, file types).
- If no suitable inputs exist locally, synthesize minimal inputs that satisfy the model contract (e.g., generated images, random tensors saved in the expected container format, short synthetic video).
- Save all chosen/generated inputs under `tmp/<experiment-dir>/inputs/`.

### 7) Run minimal, traceable inference experiments (default: inference + end-to-end timing)

- Start with a single known-good example (from upstream repo) if available.
- Save every “input → output” mapping:
  - Inputs: the exact file(s) used + preprocessing parameters.
  - Outputs: raw model outputs + any decoded/visualized artifacts.
  - Command line + environment notes (device, precision, batch size).
- Measure end-to-end timing by default:
  - At minimum: one cold run + a small number of warm runs (record mean/median).
- Persist stats that will appear in the report:
  - For any timing/profiling/memory/throughput numbers you plan to put into the report, also write a JSON version under `tmp/<experiment-dir>/outputs/` (e.g., `outputs/stats.json`).
- Capture logs by default:
  - Save stdout/stderr and command transcripts under `tmp/<experiment-dir>/logs/`.
- If the model is accessed via HTTP/gRPC, save request/response payloads (sanitized) under `reports/` and/or `outputs/`.

### 7b) (Optional) Training sanity check

If the user asks to validate training (or if inference is insufficient to validate “works”):
- Start with a minimal configuration (single batch / tiny subset) to confirm the forward + backward pass runs.
- Record key configs (optimizer, LR, batch size, mixed precision) and any dataset assumptions.
- Do not run long trainings unless the user explicitly requests it.

### 8) Produce reports

#### 8a) Ensure machine-readable report inputs exist (in `outputs/`)

Write/collect machine-readable files in `tmp/<experiment-dir>/outputs/` that the report generator can consume, at minimum:
- `stats.json` (timing/throughput/memory/profile numbers)
- A JSON describing key parameters used (preprocess/postprocess/runtime thresholds)
- A JSON describing the I/O contract (input expectations + output structure)
- A JSON listing key artifacts produced (paths to representative inputs/outputs)

Keep these JSON files as the source of truth for anything that will appear as “final stats” in the experiment report.

#### 8b) Generate `reports/experiment-report.md` programmatically

- Generate `tmp/<experiment-dir>/reports/experiment-report.md` by reading only `tmp/<experiment-dir>/outputs/` (and optionally `logs/` for pointers), with minimal/no reasoning.
- If images are part of the inputs/outputs, copy representative images into `tmp/<experiment-dir>/reports/figures/` and embed them in the markdown via relative paths (e.g., `figures/<name>.png`).

#### 8c) Write `reports/stakeholder-report.md` (agent-written)

- Read `reports/experiment-report.md` plus relevant `outputs/` and `logs/`.
- Produce `tmp/<experiment-dir>/reports/stakeholder-report.md` with deeper analysis that requires reasoning:
  - Interpret results vs expectations/targets
  - Call out risks, assumptions, and failure modes
  - Recommend next experiments and concrete integration guidance (if requested)
  - Summarize “go/no-go” criteria and what remains unknown

Also include:
- **Benchmark & profiling** results:
  - CPU/GPU model, RAM/VRAM, OS, Python version, key library versions
  - Latency breakdown if possible (preprocess / model / postprocess)
  - Throughput (items/s) and peak memory/VRAM
- **Stats JSON**:
  - For any stats included in the report, ensure the same values exist in a JSON file under `tmp/<experiment-dir>/outputs/` (e.g., `outputs/stats.json`).
- **User metrics** (if provided):
  - The metric definition + measurement method
  - Results on the chosen evaluation inputs
  - Any deltas vs the user’s targets and suggested next experiments

## Guardrails

- Do not commit large checkpoints or huge outputs; keep them under gitignored paths (`checkpoints/`, `tmp/`).
- Respect upstream licenses; record the repo URL + commit/tag in `reports/`.
- Avoid modifying runtime code under `src/` unless the user explicitly requests integration; keep exploration isolated to `tmp/<experiment-dir>`.