robotics-vla

# Robotics VLA Skill

Expert guidance for building generalist robot policies using Vision-Language-Action (VLA) flow models, based on the π0 architecture.

## Core Architecture

**π0 model = VLM backbone + action expert + flow matching**

| Component | Detail |
|---|---|
| VLM backbone | PaliGemma (3B) — provides visual + language understanding |
| Action expert | Separate transformer weights (~300M) for robot state + actions |
| Total params | ~3.3B |
| Action output | Chunks of H=50 actions; 50Hz or 20Hz robots |
| Inference speed | ~73ms on RTX 4090 |

See `references/architecture.md` for full technical details (attention masks, flow matching math, MoE design).

## Training Pipeline

**Two-phase approach (mirrors LLM training):**

1. **Pre-training** → broad physical capabilities + recovery behaviors across many tasks/robots
2. **Fine-tuning** → fluent, task-specific execution on target task

Key rule: combining both phases outperforms either alone. Pre-training gives robustness; fine-tuning gives precision.

See `references/training.md` for data mixture ratios, loss functions, and fine-tuning dataset sizing.

## Action Representation

**Use flow matching, not autoregressive discretization.**

- Flow matching models continuous action distributions → essential for high-frequency dexterous control
- Autoregressive token prediction (e.g. RT-2 style) cannot produce action chunks efficiently
- Action chunks allow open-loop execution at 50Hz without temporal ensembling

## Multi-Embodiment Support

Single model handles 7+ robot configurations via:
- Zero-padding smaller action spaces to match the largest (17-dim)
- Shared VLM backbone; embodiment-specific behavior learned via data
- Weighted task sampling: n^0.43 to handle imbalanced data across robot types

See `references/embodiments.md` for robot platform specs and action space details.

## High-Level Policy Integration

For long-horizon tasks, use a two-tier approach:
- **High-level VLM**: decomposes task ("bus the table") → subtasks ("pick up napkin")
- **Low-level π0**: executes each subtask as a language-conditioned action sequence

Analogous to SayCan. Intermediate language commands significantly boost performance vs. flat task descriptions.

## Related & Complementary Research (2025)

π0 has been extended and complemented by several key works. See `references/related-work.md` for the full landscape, including:
- **π0-FAST / π0.5 / π0.6** — direct successors with faster training, open-world generalization, and RL fine-tuning
- **RTC** — async action chunking to eliminate inference pauses (plug-in, no retraining)
- **UniVLA** — unsupervised action extraction from raw video (no action labels needed)
- **ManiFlow / Streaming Flow** — smoother action generation
- **GR00T N1, Helix, OpenVLA-OFT, DiVLA, RDT-1B** — parallel approaches from NVIDIA, Figure AI, and academia

## Evaluation Checklist

When evaluating a robot manipulation policy:
- [ ] Out-of-box generalization (no fine-tuning) vs. baselines
- [ ] Language following accuracy with flat / human-guided / HL commands
- [ ] Fine-tuning efficiency (success rate vs. hours of data)
- [ ] Complex multi-stage tasks (5–20 min, recovery from failure)
- [ ] Compare: OpenVLA, Octo, ACT, Diffusion Policy as baselines