numpy-low-level
Advanced sub-skill for NumPy focused on internal memory management, stride manipulation, structured arrays, and interfacing with C/Cython. Covers zero-copy operations and SIMD vectorization principles.
Best use case
numpy-low-level is best used when you need a repeatable AI agent workflow instead of a one-off prompt.
Advanced sub-skill for NumPy focused on internal memory management, stride manipulation, structured arrays, and interfacing with C/Cython. Covers zero-copy operations and SIMD vectorization principles.
Teams using numpy-low-level should expect a more consistent output, faster repeated execution, less prompt rewriting.
When to use this skill
- You want a reusable workflow that can be run more than once with consistent structure.
When not to use this skill
- You only need a quick one-off answer and do not need a reusable workflow.
- You cannot install or maintain the underlying files, dependencies, or repository context.
Installation
Claude Code / Cursor / Codex
Manual Installation
- Download SKILL.md from GitHub
- Place it in
.claude/skills/numpy-low-level/SKILL.mdinside your project - Restart your AI agent — it will auto-discover the skill
How numpy-low-level Compares
| Feature / Agent | numpy-low-level | Standard Approach |
|---|---|---|
| Platform Support | Not specified | Limited / Varies |
| Context Awareness | High | Baseline |
| Installation Complexity | Unknown | N/A |
Frequently Asked Questions
What does this skill do?
Advanced sub-skill for NumPy focused on internal memory management, stride manipulation, structured arrays, and interfacing with C/Cython. Covers zero-copy operations and SIMD vectorization principles.
Where can I find the source code?
You can find the source code on GitHub using the link provided at the top of the page.
SKILL.md Source
# NumPy - Low-Level Optimization & Memory
At high volumes, standard NumPy operations can still be slow due to unnecessary memory allocations. This guide covers how to manipulate the internal representation of arrays to achieve C-level performance without leaving Python.
## When to Use
- Implementing sliding window algorithms (convolutions) without extra memory.
- Interfacing Python with C, C++, or Fortran code via pointers.
- Working with complex, heterogeneous data structures (Structured Arrays).
- Optimizing memory-constrained systems via Memory Mapping (memmap).
- Debugging performance issues related to "Memory Layout" (C-style vs Fortran-style).
## Core Principles
### 1. The Metadata vs. Data Split
A NumPy array is a small Header (shape, dtype, strides) pointing to a large Data Buffer. Many operations (like `.T`, `reshape`, `slice`) only change the Header. This is "Zero-Copy".
### 2. Strides (The Step Logic)
Strides define how many bytes to skip in memory to get to the next element in each dimension. Manipulating strides allows you to "cheat" and create virtual views of data.
### 3. Contiguity
- **C-Contiguous**: Last index varies fastest (Row-major).
- **F-Contiguous**: First index varies fastest (Column-major).
- Vectorization is significantly faster on contiguous memory.
## Quick Reference: Memory Inspection
```python
import numpy as np
arr = np.zeros((100, 100))
print(arr.flags) # Check contiguity and ownership
print(arr.strides) # bytes to step in each axis
print(arr.__array_interface__['data']) # Memory pointer address
```
## Critical Rules
### ✅ DO
- **Prefer Views over Copies** - Use slicing and reshaping whenever possible.
- **Check base** - Use `arr.base is None` to verify if an array owns its memory or is just a view.
- **Use Structured Arrays** - For "Table of Records" data where you need NumPy speed but different types per column.
- **Align Memory** - Ensure arrays are aligned to 64-bit boundaries for SIMD optimization.
- **Use out= parameters** - Most NumPy functions accept an `out` argument to prevent creating a new temporary array.
### ❌ DON'T
- **Don't use np.append or np.concatenate in loops** - These are O(N²) because they copy the entire buffer every time.
- **Don't ignore the "Copy Warning"** - Fancy indexing (`arr[[1, 3, 5]]`) always creates a copy, unlike basic slicing.
- **Don't use as_strided blindly** - It is the most dangerous function in NumPy. It can lead to memory corruption or crashes if bounds are miscalculated.
## Low-Level Patterns
### 1. Sliding Windows (Zero-Copy Convolution)
```python
from numpy.lib.stride_tricks import as_strided
def sliding_window_1d(arr, window_size):
"""Creates a virtual 2D view of a 1D array for rolling stats."""
itemsize = arr.itemsize
shape = (arr.size - window_size + 1, window_size)
strides = (itemsize, itemsize)
return as_strided(arr, shape=shape, strides=strides)
# Result is a 2D array where each row is a window,
# but it uses NO additional memory.
```
### 2. Structured Arrays (Interoperable C-structs)
```python
# Define a record type: Name (32 chars), Age (int), Salary (float)
dtype = np.dtype([('name', 'S32'), ('age', 'i4'), ('salary', 'f8')])
data = np.array([('Alice', 25, 50000), ('Bob', 30, 60000)], dtype=dtype)
# Access by field name (Fast, vectorized)
print(data['salary'].mean())
```
## Interfacing with C-API
### Using ctypes pointers
```python
import ctypes
# Get raw pointer to data
ptr = arr.ctypes.data_as(ctypes.POINTER(ctypes.c_double))
# This pointer can be passed to a C/C++ library
# for direct manipulation of the NumPy buffer.
```
## Performance Optimization
### Vectorized Math with out=
```python
# ❌ SLOW: Creates 3 temporary arrays
# res = (a * b) + c
# ✅ FAST: Reuse memory
np.multiply(a, b, out=a) # a now holds a*b
np.add(a, c, out=a) # a now holds (a*b)+c
```
### Memory Mapping for Huge Data
```python
# Create an array that stays on disk, reading only what's needed
huge_data = np.memmap('data.bin', dtype='float32', mode='w+', shape=(10000, 10000))
```
## Common Pitfalls
### Broadcast Copying
If you broadcast a small array across a large one, NumPy doesn't copy the small one; it just sets the stride to 0.
```python
# strides of a (10, 1) array broadcasted to (10, 100):
# (8, 0) -> It keeps reading the same memory for the second axis!
```
### The Byte-Order (Endianness)
Scientific data from old instruments might be Big-Endian.
```python
# Convert in-place without copying
raw_data = np.frombuffer(buffer, dtype='>f4').view('<f4')
```
NumPy Low-Level is about removing abstractions. By mastering strides and the array interface, you turn Python into a thin wrapper over raw memory, enabling "impossible" data manipulations at hardware speeds.Related Skills
numpy
Comprehensive guide for NumPy - the fundamental package for scientific computing in Python. Use for array operations, linear algebra, random number generation, Fourier transforms, mathematical functions, and high-performance numerical computing. Foundation for SciPy, pandas, scikit-learn, and all scientific Python.
xgboost-lightgbm
Industry-standard gradient boosting libraries for tabular data and structured datasets. XGBoost and LightGBM excel at classification and regression tasks on tables, CSVs, and databases. Use when working with tabular machine learning, gradient boosting trees, Kaggle competitions, feature importance analysis, hyperparameter tuning, or when you need state-of-the-art performance on structured data.
xarray
N-dimensional labeled arrays and datasets in Python. Built on top of NumPy and Dask. It introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making data analysis in physical sciences more intuitive and less error-prone. Use for working with multi-dimensional scientific data, NetCDF/GRIB/Zarr files, climate/weather/oceanographic datasets, remote sensing, geospatial imaging, large out-of-memory datasets with Dask, and labeled array operations.
transformers
State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX. Provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. The industry standard for Large Language Models (LLMs) and foundation models in science.
tqdm
A fast, extensible progress bar for Python and CLI. Instantly makes your loops show a smart progress meter with ETA, iterations per second, and customizable statistics. Minimal overhead. Use for monitoring long-running loops, simulations, data processing, ML training, file downloads, I/O operations, command-line tools, pandas operations, parallel tasks, and nested progress bars.
tensorflow
Comprehensive deep learning framework for building, training, and deploying neural networks. TensorFlow provides tf.keras high-level API for model construction, tf.data for efficient data pipelines, and tf.function for graph-mode optimization. Use when working with: neural network training and inference, image classification/detection/segmentation, NLP/text processing with embeddings or transformers, time series forecasting, generative models (VAE, GAN), transfer learning with pretrained models, custom training loops with GradientTape, GPU/TPU accelerated computation, or any deep learning task.
sympy
Comprehensive guide for SymPy - Python library for symbolic mathematics. Use for symbolic expressions, calculus (derivatives, integrals, limits, series), equation solving (algebraic, differential, systems), linear algebra, simplification, matrix operations, special functions, code generation, and mathematical proofs. Essential for analytical mathematics and computer algebra.
sunpy
The community-developed free and open-source software package for solar physics. Provides tools for data search and download, coordinate transformations specific to solar physics, and powerful image processing through the Map object. Use when working with solar data, solar images (EUV, magnetograms, white light), solar coordinates (Helioprojective, Heliographic), Fido data search, solar time series, differential rotation, limb fitting, or multi-instrument solar analysis (AIA, HMI, GOES).
statsmodels
Advanced statistical modeling and hypothesis testing. Complementary to SciPy's stats module, it provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests and statistical data exploration. Use for linear regression, GLM, time series analysis, ANOVA, survival analysis, causal inference, and statistical hypothesis testing. Load when working with OLS, WLS, logistic regression, Poisson regression, ARIMA, SARIMAX, statistical diagnostics, p-values, confidence intervals, or R-style statistical analysis.
spacy-nltk
Natural Language Processing for text analysis, corpus linguistics, and production NLP pipelines. spaCy provides fast production-grade tokenization, POS tagging, NER, dependency parsing, and custom model training. NLTK provides classical corpus linguistics, linguistic analysis, VADER sentiment, collocation analysis, and access to standard linguistic corpora. Use when: processing and analyzing text data, extracting named entities (people, orgs, locations, dates), dependency parsing and syntactic analysis, building text classification pipelines, performing corpus-level linguistic analysis (frequency, collocations, readability), sentiment analysis, lemmatization and stemming, working with multilingual text, training custom NER or text classifiers, or any task requiring structured understanding of natural language beyond simple string operations.
sktime-tsfresh
Time series machine learning layer (Tier 1): integration of **sktime** and **tsfresh** for building production-grade pipelines that transform raw time series into tabular feature representations suitable for classical machine-learning models. *sktime* provides a unified, sklearn-compatible interface for time-series data types, transformations, and pipelines, while *tsfresh* enables large-scale automated extraction of statistical, spectral, and autocorrelation features, with optional statistically grounded feature relevance selection (FRESH).
sklearn-explainability
Advanced sub-skill for scikit-learn focused on model interpretability, feature importance, and diagnostic tools. Covers global and local explanations using built-in inspection tools and SHAP/LIME integrations.