# Mapping Gemma3 onto an Edge Dataflow Architecture

arXiv: https://arxiv.org/abs/2602.06063 · HTML: https://mlsanity.com/paper/2602.06063/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 6 of 6)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Online normalizer calculation for softmax](https://mlsanity.com/paper/1805.02867/) (2018)
- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)
- [Efficiency, Expressivity, and Extensibility in a Close-to-Metal NPU Programming Interface](https://mlsanity.com/paper/2504.18430/) (2025)
- [Unlocking the AMD Neural Processing Unit for ML Training on the Client Using Bare-Metal-Programming Tools](https://mlsanity.com/paper/2504.03083/) (2025)

## Built on by (most cited first, 2 of 2)

- [TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs](https://mlsanity.com/paper/2606.11357/) (2026)
- [Programming AMD XDNA NPUs with Open-source Compiler Tools: A FlashAttention Case Study](https://mlsanity.com/paper/2609.21264/) (2026)

## Abstract (arXiv)

We present the first end-to-end deployment of the Gemma3 family of large language and vision models on a tiled edge dataflow architecture (AMD Ryzen AI NPU). Our work introduces a set of hardware-aware techniques. For prefill, we introduce an efficient dequantization engine, optimize tiled matrix multiplication kernels, and propose FlowQKV, a chunked, pipelined attention mechanism. For decoding, we introduce FusedDQP, which fuses dequantization and projection into a single kernel, and FlowKV, which re-structures attention to sustain high memory bandwidth utilization. Together with a compact Q4NX 4-bit quantization format, these methods yield up to 5.2× faster prefill and 4.8× faster decoding versus the iGPU, and 33.5× and 2.2× over the CPU, respectively. Power efficiency improves by as much as 67.2× and 222.9× compared to the iGPU and CPU. The proposed approach demonstrates that modern NPUs can deliver practical, low-power LLM and VLM inference at the edge, and provides a generalizable blueprint for mapping transformer-based models onto tiled dataflow accelerators.