mlsanityData to 7 Oct 2026

Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions

arXiv 2510.22909 ↗First posted 27 Oct 2025cs.DC cs.AI cs.PFcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Edge intelligent applications like VR/AR and language model based chatbots have become widespread with the rapid expansion of IoT and mobile devices. However, constrained edge devices often cannot serve the increasingly large and complex deep learning (DL) models. To mitigate these challenges, researchers have proposed optimizing and offloading partitions of DL models among user devices, edge servers, and the cloud. In this setting, users can take advantage of different services to support their intelligent applications. For example, edge resources offer low response latency. In contrast, cloud platforms provide low monetary cost computation resources for computation-intensive workloads. However, communication between DL model partitions can introduce transmission bottlenecks and pose risks of data leakage. Recent research aims to balance accuracy, computation delay, transmission delay, and privacy concerns. They address these issues with model compression, model distillation, transmission compression, and model architecture adaptations, including internal classifiers. This survey contextualizes the state-of-the-art model offloading methods and model adaptation techniques by studying their implication to a multi-objective optimization comprising inference latency, data privacy, and resource monetary cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 91 of its 191 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

91 of 191 references.

References · 91

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2018YOLOv3: An Incremental Improvementcited by 2,652
38 more
  1. 2019Searching for MobileNetV3cited by 1,984
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  4. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  13. 2022Efficiently Scaling Transformer Inferencecited by 529
  14. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  15. 2021Dynamic Neural Networks: A Surveycited by 405
  16. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  17. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  18. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  19. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  20. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  21. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  22. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  23. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  24. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  25. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  26. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  27. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  28. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  29. 2018Gazelle: A Low Latency Framework for Secure Neural Network Inferencecited by 193
  30. 2015Inverting Visual Representations with Convolutional Networkscited by 168
  31. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  32. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  33. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  34. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
  35. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 119
  36. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  37. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  38. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109

50 most cited of 91 All 91 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  2. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  3. 2026Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
  4. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deployment
  5. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
  6. 2026Vision-Language Models on the Edge for Real-Time Robotic Perception
  7. 2026A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
  8. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices