# RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis

arXiv: https://arxiv.org/abs/2602.11506 · HTML: https://mlsanity.com/paper/2602.11506/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [A Review on Edge Large Language Models: Design, Execution, and Applications](https://mlsanity.com/paper/2410.11845/) (2024)
- [Densing Law of LLMs](https://mlsanity.com/paper/2412.04315/) (2024)
- [Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS](https://mlsanity.com/paper/2511.05502/) (2025)
- [PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing](https://mlsanity.com/paper/2503.12167/) (2025)
- [Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends](https://mlsanity.com/paper/2511.22334/) (2025)
- [GTA: Grouped-head latenT Attention](https://mlsanity.com/paper/2506.17286/) (2025)

## Built on by (most cited first, 5 of 5)

- [Gated Subspace Inference for Transformer Acceleration](https://mlsanity.com/paper/2605.03109/) (2026)
- [Think Before You Grid-Search: Floor-First Triage for LLM Serving](https://mlsanity.com/paper/2607.05876/) (2026)
- [Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising](https://mlsanity.com/paper/2607.24841/) (2026)
- [GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems](https://mlsanity.com/paper/2609.14864/) (2026)
- [Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design](https://mlsanity.com/paper/2610.05062/) (2026)

## Abstract (arXiv)

The transition toward localized intelligence through Small Language Models (SLMs) has intensified the need for rigorous performance characterization on resource-constrained edge hardware. However, objectively measuring the theoretical performance ceilings of diverse architectures across heterogeneous platforms remains a formidable challenge. In this work, we propose a systematic framework based on the Roofline model that unifies architectural primitives and hardware constraints through the lens of operational intensity (OI). By defining an inference-potential region, we introduce the Relative Inference Potential as a novel metric to compare efficiency differences between Large Language Models (LLMs) on the same hardware substrate. Extensive empirical analysis across diverse compute tiers reveals that variations in performance and OI are significantly influenced by sequence length. We further identify a critical regression in OI as model depth increases. Additionally, our findings highlight an efficiency trap induced by hardware heterogeneity and demonstrate how structural refinements, such as Multi-head Latent Attention (MLA), can effectively unlock latent inference potential across various hardware substrates. These insights provide actionable directions for hardware-software co-design to align neural structures with physical constraints in on-device intelligence. The released code is available in the Appendix C.