hot3d

# HOT3D - Multi-View 3D Hand & Object Tracking

## Overview

State-of-the-art multi-view 3D tracking system for egocentric hand-object interactions from Meta Facebook Research. Designed for Aria smart glasses and Quest VR headsets, HOT3D provides precise 3D world coordinates for hand joints, manipulated objects, and their interactions. The system includes visualization tools for rendering 3D overlays on video frames with joint projections, hand meshes, and object models.

**Project page**: https://facebookresearch.github.io/hot3d

**Best for**: Research-grade 3D tracking with multi-camera setups, high-precision applications, and XR device integration.

## When to Use This Skill

Use when you need:
- **Multi-view 3D tracking** with world coordinates
- **High-precision hand pose** in 3D space (millimeter accuracy)
- **Object tracking** during manipulation
- **Aria/Quest integration** for wearable devices
- **Research-grade tracking** benchmarks
- **Hand-object interaction** analysis in 3D

**vs alternatives**:
- More advanced than single-view methods (hands-3d-pose)
- Higher precision than bounding box detection (handtracking)
- Full 3D world coordinates vs 2D projections

## Core Capabilities

### 1. Multi-View 3D Hand Tracking

**21-keypoint 3D hand pose** from multiple synchronized cameras:
- 3D world coordinates (x, y, z) for each joint
- Joint confidence scores
- Left/right hand identification
- Temporal consistency across frames
- Hand mesh reconstruction

### 2. Object Pose Estimation

**6DOF object pose** tracking:
- 3D position and orientation (quaternion/rotation matrix)
- Object mesh alignment
- Tracking during manipulation
- Multiple object support

### 3. Hand-Object Interaction

**Interaction analysis**:
- Contact point detection
- Grasp type classification
- Manipulation phase detection
- Force estimation (with sensor data)

### 4. Visualization Tools

**Rich visualization options**:
- 3D skeleton projected to each camera view
- Hand mesh rendering
- Object model overlay
- Trajectory visualization
- Multi-view synchronized display

## Quick Start

```bash
# Clone repository
git clone https://github.com/facebookresearch/hot3d.git
cd hot3d

# Install dependencies
pip install -r requirements.txt
# Key: PyTorch3D, Open3D, vispy

# Download dataset (requires registration)
# https://facebookresearch.github.io/hot3d/dataset.html

# Run demo
python demo/visualize_tracking.py \
    --sequence demo_sequence \
    --output_dir ./visualizations
```

## Usage Example

```python
from hot3d import HOT3DTracker
import numpy as np

# Initialize tracker
tracker = HOT3DTracker()
tracker.load_sequence('path/to/sequence')

# Get frame data
frame_data = tracker.get_frame(frame_id=100)

# Access 3D hand pose
hand_pose_3d = frame_data['left_hand']  # 21x3 array
print(f"Wrist position: {hand_pose_3d[0]}")  # [x, y, z]

# Access object pose
object_pose = frame_data['object_001']
position = object_pose['position']  # [x, y, z]
rotation = object_pose['rotation']  # 3x3 matrix

# Visualize
tracker.visualize_frame(
    frame_id=100,
    show_hands=True,
    show_objects=True,
    show_meshes=True,
    save_path='output.png'
)
```

## Model Specs

- **Input**: Multi-view RGB-D video streams (typically 3-5 cameras)
- **Output**: 3D coordinates in world frame (millimeters)
- **Accuracy**: ~5-10mm hand joint error
- **Frame rate**: 30-60 Hz (depends on hardware)
- **Latency**: <100ms for real-time applications

## Requirements

- **Hardware**: Multi-camera setup or Aria/Quest device
- **Computation**: GPU recommended (NVIDIA RTX 3080 or better)
- **Storage**: Large dataset (several TB for full HOT3D)
- **Software**: PyTorch, PyTorch3D, Open3D

## Dataset

**HOT3D dataset** includes:
- 100+ sequences of daily activities
- Multi-view RGB-D video
- 3D hand and object annotations
- Aria/Quest recordings
- Smart glasses data

Access: https://facebookresearch.github.io/hot3d

## Integration

Works with:
- **hand-tracking-toolkit**: Evaluation and metrics
- **Aria SDK**: Device integration
- **PyTorch3D**: 3D processing
- **OpenXR**: XR platform integration

## Limitations

- Requires multi-view setup or specialized hardware
- Computational intensive
- Dataset access requires registration
- Complex setup compared to single-view methods

## Best For

- **XR applications** with smart glasses
- **Research** in 3D hand tracking
- **High-precision** manipulation analysis
- **Benchmarking** new algorithms

## References

- Project: https://facebookresearch.github.io/hot3d
- GitHub: https://github.com/facebookresearch/hot3d
- Paper: HOT3D dataset publication
- Citation: See project page

## License

CC-BY-NC 4.0 (non-commercial only)