mlsanityData to 14 Oct 2026

Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

arXiv 2505.16508 ↗First posted 22 May 2025cs.DC cs.AI cs.NIcited by 13

Not on the map. Why?

Abstract (arXiv)

The widespread adoption of Language Models (LMs) across industries is driving interest in deploying these services across the computing continuum, from the cloud to the network edge. This shift aims to reduce costs, lower latency, and improve reliability and privacy. Small Language Models (SLMs), enabled by advances in model compression, are central to this shift, offering a path to on-device inference on resource-constrained edge platforms. This work examines the interplay between edge and cloud deployments, starting from detailed benchmarking of SLM capabilities on single edge devices, and extending to distributed edge clusters. We identify scenarios where edge inference offers comparable performance with lower costs, and others where cloud fallback becomes essential due to limits in scalability or model capacity. Rather than proposing a one-size-fits-all solution, we present platform-level comparisons and design insights for building efficient, adaptive LM inference systems across heterogeneous environments.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 11 of its 21 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

11 of 21 references.

References · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  7. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  8. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  9. 2024Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Datacited by 66
  10. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  11. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
  2. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  3. 2025ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Modelscited by 4
  4. 2026Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inferencecited by 2
  5. 2026BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metalcited by 1
  6. 2026LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusterscited by 1
  7. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levelscited by 1
  8. 2026Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systemscited by 1
  9. 2025Beyond Manually Designed Pruning Policies with Second-Level Performance Prediction: A Pruning Framework for LLMscited by 0
  10. 2026AI-Enabled Covert Channel Detection in RF Receiver Architecturescited by 0
  11. 2026BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Acceleratorscited by 0
  12. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edgecited by 0
1 more
  1. 2026DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environmentscited by 0