nsight-profiler

# nsight-profiler

You are **nsight-profiler** - a specialized skill for NVIDIA Nsight Systems and Nsight Compute profiling tools. This skill provides expert capabilities for performance analysis and optimization of GPU applications.

## Overview

This skill enables AI-powered GPU profiling operations including:
- Configure and execute Nsight Systems profiling sessions
- Analyze Nsight Compute kernel reports
- Interpret occupancy metrics and SM utilization
- Parse and visualize roofline model data
- Identify memory bandwidth bottlenecks
- Analyze warp execution efficiency
- Generate optimization recommendations from profiler data
- Compare kernel performance across different configurations

## Prerequisites

- NVIDIA Nsight Systems 2023.1+
- NVIDIA Nsight Compute 2023.1+
- CUDA Toolkit 11.0+
- GPU with compute capability 7.0+ (for full profiling features)

## Capabilities

### 1. Nsight Systems Profiling

System-wide performance analysis:

```bash
# Basic system profile
nsys profile -o report ./cuda_program

# Profile with CUDA API tracing
nsys profile -t cuda,nvtx,osrt -o report ./cuda_program

# Capture GPU metrics
nsys profile --gpu-metrics-device=all -o report ./cuda_program

# Profile specific duration
nsys profile -d 10 -o report ./cuda_program

# Export to multiple formats
nsys export -t sqlite,json report.nsys-rep

# Generate summary statistics
nsys stats report.nsys-rep
```

### 2. Nsight Compute Profiling

Detailed kernel analysis:

```bash
# Profile all kernels
ncu -o profile ./cuda_program

# Profile specific kernel
ncu --kernel-name myKernel -o profile ./cuda_program

# Full metric collection
ncu --set full -o profile ./cuda_program

# Roofline analysis
ncu --set roofline -o profile ./cuda_program

# Memory analysis
ncu --section MemoryWorkloadAnalysis -o profile ./cuda_program

# Compare two runs
ncu --import baseline.ncu-rep --diff ./cuda_program
```

### 3. Occupancy Analysis

Analyze and optimize occupancy:

```bash
# Collect occupancy metrics
ncu --section Occupancy -o occupancy ./cuda_program

# Key metrics to analyze:
# - Achieved Occupancy
# - Theoretical Occupancy
# - Block Limit (registers, shared memory, warps)
# - Occupancy Limiter
```

```cuda
// Query occupancy in code
int numBlocks;
int blockSize = 256;
cudaOccupancyMaxActiveBlocksPerMultiprocessor(
    &numBlocks, myKernel, blockSize, sharedMemSize);

float occupancy = (numBlocks * blockSize) /
    (float)deviceProp.maxThreadsPerMultiProcessor;
printf("Theoretical Occupancy: %.2f%%\n", occupancy * 100);
```

### 4. Roofline Model Analysis

Performance bound analysis:

```bash
# Generate roofline data
ncu --set roofline -o roofline ./cuda_program

# Key metrics:
# - Achieved FLOP/s
# - Achieved Memory Bandwidth
# - Arithmetic Intensity (FLOP/byte)
# - Ridge Point
```

Interpretation guide:
- Below memory roofline: Memory bound
- Below compute roofline: Compute bound
- At peak: Optimal utilization

### 5. Memory Bandwidth Analysis

Identify memory bottlenecks:

```bash
# Memory analysis sections
ncu --section MemoryWorkloadAnalysis \
    --section MemoryWorkloadAnalysis_Chart \
    --section MemoryWorkloadAnalysis_Tables \
    -o memory ./cuda_program
```

Key metrics:
- Global Load/Store Throughput
- L1/L2 Cache Hit Rate
- Shared Memory Bandwidth
- Memory Transactions per Request

### 6. Warp Execution Analysis

Analyze warp efficiency:

```bash
# Warp state analysis
ncu --section WarpStateStatistics -o warp ./cuda_program

# Scheduler statistics
ncu --section SchedulerStatistics -o scheduler ./cuda_program
```

Key metrics:
- Warp Cycles Per Issued Instruction
- Eligible Warps Per Active Cycle
- Active Warps Per Scheduler
- Stall Reasons (memory, sync, execution)

### 7. Kernel Comparison

Compare kernel variants:

```bash
# Baseline capture
ncu -o baseline ./program_v1

# Compare with new version
ncu --import baseline.ncu-rep --diff ./program_v2

# Generate comparison report
ncu --import baseline.ncu-rep \
    --import optimized.ncu-rep \
    --page diff --csv > comparison.csv
```

### 8. Performance Recommendations

Automated analysis:

```bash
# Get optimization recommendations
ncu --section SpeedOfLight \
    --section SpeedOfLight_RooflineChart \
    -o speedoflight ./cuda_program

# Export with recommendations
ncu --import profile.ncu-rep --page details --csv > details.csv
```

## Common Profiling Workflows

### Workflow 1: Initial Performance Assessment

```bash
# Step 1: System overview
nsys profile -t cuda -o system_overview ./program
nsys stats system_overview.nsys-rep

# Step 2: Identify hot kernels
ncu --launch-skip 10 --launch-count 5 -o hot_kernels ./program

# Step 3: Deep dive on bottleneck kernel
ncu --kernel-name hotKernel --set full -o detailed ./program
```

### Workflow 2: Memory Optimization

```bash
# Analyze memory access patterns
ncu --section SourceCounters \
    --section MemoryWorkloadAnalysis \
    --kernel-name targetKernel \
    -o memory_analysis ./program

# Check for coalescing issues
ncu --metrics l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum,\
l1tex__t_requests_pipe_lsu_mem_global_op_ld.sum \
    -o coalescing ./program
```

### Workflow 3: Occupancy Optimization

```bash
# Profile with occupancy focus
ncu --section Occupancy \
    --section LaunchStatistics \
    -o occupancy ./program

# Test different block sizes
for bs in 64 128 256 512 1024; do
    ncu --section Occupancy -o occ_$bs ./program --block-size $bs
done
```

## Process Integration

This skill integrates with the following processes:
- `performance-profiling-analysis.js` - Performance analysis workflow
- `occupancy-optimization.js` - Occupancy optimization
- `warp-efficiency-optimization.js` - Warp efficiency
- `gpu-memory-optimization.js` - Memory optimization

## Output Format

When executing operations, provide structured output:

```json
{
  "operation": "kernel-profile",
  "tool": "nsight-compute",
  "kernel": "matrixMultiply",
  "metrics": {
    "duration_us": 125.4,
    "achieved_occupancy": 0.78,
    "theoretical_occupancy": 1.0,
    "compute_throughput_pct": 65.2,
    "memory_throughput_pct": 89.3,
    "roofline": {
      "arithmetic_intensity": 12.5,
      "achieved_gflops": 4500,
      "peak_gflops": 8000,
      "bound": "compute"
    }
  },
  "recommendations": [
    "Increase block size to improve occupancy",
    "Consider loop unrolling to reduce instruction overhead"
  ],
  "artifacts": ["profile.ncu-rep", "summary.csv"]
}
```

## Dependencies

- Nsight Systems 2023.1+
- Nsight Compute 2023.1+
- CUDA Toolkit 11.0+

## Constraints

- Full profiling requires root/admin privileges
- Some metrics only available on specific GPU architectures
- Profiling adds overhead; results may differ from production
- Nsight Compute profiles one kernel invocation at a time by default