mlsanityData to 14 Oct 2026

CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation

arXiv 2510.19670 ↗First posted 22 Oct 2025cs.CLcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

We present CoSense-LLM, an edge-first framework that turns continuous multimodal sensor streams (for example Wi-Fi CSI, IMU, audio, RFID, and lightweight vision) into compact, verifiable semantic tokens and coordinates with large language models under explicit latency, energy, bandwidth, and privacy constraints. CoSense-LLM has four parts: (i) SenseFusion, a lightweight encoder that aligns sensor embeddings with language and compresses them into short discrete code sequences; (ii) Edge-RAG, a local hybrid retrieval layer that grounds generation in site specific policies and notes; (iii) PromptRouter, a cost and uncertainty aware policy that selects edge only generation, edge plus retrieval, or compact cloud escalation; and (iv) Secure Execution, an auditable redaction path that enforces data minimization so raw waveforms never leave the device. The system works with modern serving optimizations, including paged or streaming KV caches, FlashAttention style kernels, speculative decoding, and quantized LoRA adapters, and supports on device personalization and federated updates under non IID drift. Across home, office, and clinic deployments, CoSense-LLM delivers grounded explanations while meeting tight service level objectives: it sustains sub second (p95) end to end latency on edge dominant paths, reduces inter tier token and bandwidth costs by preferring local retrieval grounded responses, and preserves privacy by transmitting only discrete codes and redacted metadata. Ablations show that Edge-RAG improves factual consistency and reduces contradictions, calibrated uncertainty enables selective abstention and controlled escalations, and KV plus decoding accelerators lower energy per decision. The results support an edge first design that treats semantics, privacy, and predictable latency as co equal goals for large model deployments in interference prone environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 78 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

47 of 78 references.

References · 47

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
35 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  6. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  7. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  9. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  10. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  11. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  12. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  13. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  14. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  15. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  16. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  17. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  18. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  19. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  20. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  21. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  22. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  23. 2025Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimizationcited by 19
  24. 2024WiOpen: A Robust Wi-Fi-based Open-set Gesture Recognition Frameworkcited by 16
  25. 2025CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edgecited by 16
  26. 2023ReSup: Reliable Label Noise Suppression for Facial Expression Recognitioncited by 14
  27. 2024EdgeRAG: Online-Indexed RAG for Edge Devicescited by 13
  28. 2024Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architecturescited by 12
  29. 2025FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuningcited by 10
  30. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  31. 2024Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimizationcited by 7
  32. 2025Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuningcited by 7
  33. 2025Towards Communication-Efficient Decentralized Federated Graph Learning over Non-IID Datacited by 6
  34. 2017Wi-Fi CSI based Behavior Recognition: From Signals, Actions to Activitiescited by 4
  35. 2023Sensing Assisted Communication for the Next Generation mmWave WLAN: System Simulation Perspectivecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Transferable Deployment of Semantic Edge Inference Systems via Unsupervised Domain Adaption
  2. 2026DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching
  3. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  4. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits
  5. 2025Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks
  6. 2026Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models
  7. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
  8. 2026WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference