# Accuracy-Delay Trade-Off in LLM Offloading via Token-Level Uncertainty

arXiv: https://arxiv.org/abs/2602.07958 · HTML: https://mlsanity.com/paper/2602.07958/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Builds on (most cited first, 12 of 12)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice](https://mlsanity.com/paper/2410.15326/) (2024)
- [Mobile Edge Computing: A Survey on Architecture and Computation Offloading](https://mlsanity.com/paper/1702.05309/) (2017)
- [Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection](https://mlsanity.com/paper/2405.02134/) (2024)
- [Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models](https://mlsanity.com/paper/2412.12687/) (2024)
- [Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading](https://mlsanity.com/paper/2501.14205/) (2025)
- [Adaptive Resource Allocation Optimization Using Large Language Models in Dynamic Wireless Environments](https://mlsanity.com/paper/2502.02287/) (2025)
- [Distributed Task Offloading and Resource Allocation for Latency Minimization in Mobile Edge Computing Networks](https://mlsanity.com/paper/2404.03154/) (2024)
- [Recursive Offloading for LLM Serving in Multi-tier Networks](https://mlsanity.com/paper/2505.16502/) (2025)
- [Joint Optimization of User Association and Resource Allocation for Load Balancing With Multi-Level Fairness](https://mlsanity.com/paper/2505.08573/) (2025)

## Abstract (arXiv)

Large language models (LLMs) offer significant potential for intelligent mobile services but are computationally intensive for resource-constrained devices. Mobile edge computing (MEC) allows such devices to offload inference tasks to edge servers (ESs), yet introduces latency due to communication and serverside queuing, especially in multi-user environments. In this work, we propose an uncertainty-aware offloading framework that dynamically decides whether to perform inference locally or offload it to the ES, based on token-level uncertainty and resource constraints. We define a margin-based token-level uncertainty metric and demonstrate its correlation with model accuracy. Leveraging this metric, we design a greedy offloading algorithm (GOA) that minimizes delay while maintaining accuracy by prioritizing offloading for highuncertainty queries. Our experiments show that GOA consistently achieves a favorable trade-off, outperforming baseline strategies in both accuracy and latency across varying user densities, and operates with practical computation time. These results establish GOA as a scalable and effective solution for LLM inference in MEC environments.