# CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge

arXiv: https://arxiv.org/abs/2506.02847 · HTML: https://mlsanity.com/paper/2506.02847/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2025.

## Built on by (most cited first, 16 of 16)

- [AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agents](https://mlsanity.com/paper/2602.06485/) (2026)
- [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRL](https://mlsanity.com/paper/2512.23310/) (2025)
- [Performance Trade-offs of Optimizing Small Language Models for E-Commerce](https://mlsanity.com/paper/2510.21970/) (2025)
- [LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference](https://mlsanity.com/paper/2601.09258/) (2026)
- [ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving](https://mlsanity.com/paper/2604.09722/) (2026)
- [CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation](https://mlsanity.com/paper/2510.19670/) (2025)
- [Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions](https://mlsanity.com/paper/2510.22909/) (2025)
- [POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Serving](https://mlsanity.com/paper/2604.16583/) (2026)
- [Communication-Efficient Verifiable Attention for LLM Inference](https://mlsanity.com/paper/2606.16352/) (2026)
- [FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs](https://mlsanity.com/paper/2607.21624/) (2026)
- [FlashAttention for Scalable Vector Architectures](https://mlsanity.com/paper/2608.18656/) (2026)
- [A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware](https://mlsanity.com/paper/2609.08307/) (2026)
- [ARID: A Deployable Edge AI System for Structured Information Extraction from Industrial Maintenance Work Orders](https://mlsanity.com/paper/2609.23582/) (2026)
- [Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design](https://mlsanity.com/paper/2610.05062/) (2026)

## Abstract (arXiv)

Deploying large language models (LLMs) on edge devices is crucial for delivering fast responses and ensuring data privacy. However, the limited storage, weight, and power of edge devices make it difficult to deploy LLM-powered applications. These devices must balance latency requirements with energy consumption and model accuracy. In this paper, we first quantify the challenges of deploying LLMs on off-the-shelf edge devices and then we present CLONE, an in-depth algorithm-hardware co-design at both the model- and system-level that intelligently integrates real-time, energy optimization while maintaining robust generality. In order to maximize the synergistic benefits of these algorithms in always-on and intermediate edge computing settings, we specialize in a 28nm scalable hardware accelerator system. We implement and extensively evaluate CLONE on two off-the-shelf edge platforms. Experiments show that CLONE effectively accelerates the inference process up to 11.92x, and saves energy up to 7.36x, while maintaining high-generation.