NEON is ARM's SIMD (single instruction, multiple data) extension — it lets a CPU core apply the same arithmetic operation to several values in one instruction instead of one. Neural network inference is full of exactly this kind of repetitive math, which makes NEON one of the highest-leverage targets for XNNPACK's hand-written kernels.
Why generic code isn't enough
A compiler can auto-vectorize simple loops to some degree, but convolution and matrix-multiply code in real models has irregular memory access patterns that compilers generally can't vectorize well on their own. XNNPACK's NEON kernels are written and tuned by hand for these specific patterns, packing data into NEON registers deliberately rather than hoping the compiler figures it out.
Where NEON kernels get used
- Depthwise and standard convolutions — the most compute-heavy operators in most vision models
- Fully connected (dense) layers, common in the final stages of classification models
- Pooling and activation functions, which are simpler but still benefit from batched execution
- Quantized INT8 operator variants, which pack more values per register than FP32
NEON vs. NEON-FP16
Newer ARM chips support a half-precision floating-point NEON extension, which roughly doubles how many values fit in a register compared to standard FP32 NEON. XNNPACK includes kernels for this variant and selects it automatically on hardware that supports it, trading some numerical precision for throughput.
Runtime kernel selection
Rather than requiring you to pick a NEON variant at build time, XNNPACK detects available CPU features at runtime and dispatches to the fastest kernel your specific device actually supports. This is why the same compiled library can run efficiently across a wide range of ARM chips with different NEON capabilities, from a mid-range phone to a newer flagship.
What this means for your build
You don't need to write NEON assembly yourself to benefit from any of this — it's already built into XNNPACK. What you do control is making sure your build doesn't accidentally restrict the instruction sets XNNPACK is allowed to use; see our guide on CMake flags worth knowing for how build-time scope settings interact with this.
Where this shows up in practice
If you're integrating through TensorFlow Lite, this NEON optimization work is exactly what's running behind the scenes when the XNNPACK delegate is active — see our TensorFlow Lite delegate guide for how that connection works.