XNNPACK's kernels are already optimized at the instruction level, so most latency gains left on the table for a mobile app come from how the model and thread pool are configured around it, not from the library itself. Here's where to look.
Thread count
XNNPACK executes through a configurable thread pool. More threads isn't automatically better on a phone — mobile CPUs mix high-performance and efficiency cores, and oversubscribing threads across all of them can cause scheduling contention that increases latency instead of reducing it. Testing a couple of thread-count settings on your actual target devices, rather than assuming "more is faster," typically finds a better number than the default.
Model precision
Quantized INT8 models generally run faster on XNNPACK's mobile kernels than FP32 equivalents, at some cost to numerical precision. For models where a small accuracy trade-off is acceptable — many vision and audio classification tasks — quantization is often the single biggest latency win available, ahead of any build-flag tuning.
Input and output memory layout
XNNPACK's kernels are tuned for specific tensor memory layouts. Feeding data in an unexpected layout can force extra reformatting passes before the actual computation starts. Checking that your preprocessing pipeline produces tensors in the layout your framework expects avoids paying that reformatting cost on every single inference call.
Warm-up runs
The first inference call after a model loads is often slower than subsequent ones, partly due to memory allocation and any lazy initialization inside the delegate. Running one or two throwaway inference passes right after loading a model — before timing anything — gives a more representative picture of steady-state latency.
Batch size
Most mobile inference is single-sample (batch size 1), which is what XNNPACK's mobile kernels are primarily tuned for. If your app somehow ends up batching multiple inputs together on-device, verify that's actually helping — on constrained mobile hardware, larger batches can increase peak memory pressure without a proportional latency benefit.
Confirming the delegate is actually engaged
None of the above matters if XNNPACK isn't actually handling your model's operators in the first place. If you're on TensorFlow Lite, see our guide on how the XNNPACK delegate works for how to confirm it's active before spending time on other tuning.