# Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

arXiv: https://arxiv.org/abs/2505.16508 · HTML: https://mlsanity.com/paper/2505.16508/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 11 of 11)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [LLMCad: Fast and Scalable On-device Large Language Model Inference](https://mlsanity.com/paper/2309.04255/) (2023)
- [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024)

## Built on by (most cited first, 13 of 13)

- [SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving](https://mlsanity.com/paper/2506.09397/) (2025)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models](https://mlsanity.com/paper/2510.27256/) (2025)
- [Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference](https://mlsanity.com/paper/2607.09520/) (2026)
- [BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal](https://mlsanity.com/paper/2607.00501/) (2026)
- [LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters](https://mlsanity.com/paper/2607.17175/) (2026)
- [Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels](https://mlsanity.com/paper/2608.25053/) (2026)
- [Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems](https://mlsanity.com/paper/2609.13559/) (2026)
- [Beyond Manually Designed Pruning Policies with Second-Level Performance Prediction: A Pruning Framework for LLMs](https://mlsanity.com/paper/2508.02381/) (2025)
- [AI-Enabled Covert Channel Detection in RF Receiver Architectures](https://mlsanity.com/paper/2604.14987/) (2026)
- [BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators](https://mlsanity.com/paper/2607.19438/) (2026)
- [BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge](https://mlsanity.com/paper/2608.05926/) (2026)
- [DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments](https://mlsanity.com/paper/2609.00442/) (2026)

## Abstract (arXiv)

The widespread adoption of Language Models (LMs) across industries is driving interest in deploying these services across the computing continuum, from the cloud to the network edge. This shift aims to reduce costs, lower latency, and improve reliability and privacy. Small Language Models (SLMs), enabled by advances in model compression, are central to this shift, offering a path to on-device inference on resource-constrained edge platforms. This work examines the interplay between edge and cloud deployments, starting from detailed benchmarking of SLM capabilities on single edge devices, and extending to distributed edge clusters. We identify scenarios where edge inference offers comparable performance with lower costs, and others where cloud fallback becomes essential due to limits in scalability or model capacity. Rather than proposing a one-size-fits-all solution, we present platform-level comparisons and design insights for building efficient, adaptive LM inference systems across heterogeneous environments.