# Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference

arXiv: https://arxiv.org/abs/2503.12926 · HTML: https://mlsanity.com/paper/2503.12926/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 52 of its 77 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)

## Built on by (most cited first, 18 of 18)

- [AI Flow: Perspectives, Scenarios, and Approaches](https://mlsanity.com/paper/2506.12479/) (2025)
- [Generative Video Compression: Towards 0.01% Compression Rate for Video Transmission](https://mlsanity.com/paper/2512.24300/) (2025)
- [SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models](https://mlsanity.com/paper/2508.03351/) (2025)
- [The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Models](https://mlsanity.com/paper/2512.23340/) (2025)
- [Theoretical Foundations of Scaling Law in Familial Models](https://mlsanity.com/paper/2512.23407/) (2025)
- [Token-level Data Selection for Safe LLM Fine-tuning](https://mlsanity.com/paper/2603.01185/) (2026)
- [Enhancing Neural Video Compression of Static Scenes with Positive-Incentive Noise](https://mlsanity.com/paper/2603.06095/) (2026)
- [Ruyi2 Technical Report](https://mlsanity.com/paper/2602.22543/) (2026)
- [Collaborative Edge-to-Server Inference for Vision-Language Models](https://mlsanity.com/paper/2512.16349/) (2025)
- [A Real-Time Privacy-Preserving Behavior Recognition System via Edge-Cloud Collaboration](https://mlsanity.com/paper/2601.22938/) (2026)
- [Task-Oriented Communication for Human Action Understanding via Edge-Cloud Co-Inference](https://mlsanity.com/paper/2605.07354/) (2026)
- [Towards Large Model Feature Coding](https://mlsanity.com/paper/2605.24025/) (2026)
- [EcoVLA: Energy-Efficient Device-Edge Co-Inference for Vision-Language-Action Models under Real-Time Constraints](https://mlsanity.com/paper/2608.15502/) (2026)
- [ISCS: Parameter-Guided Feature Pruning for Resource-Constrained Embodied Perception](https://mlsanity.com/paper/2509.16853/) (2025)
- [Privacy-Aware Camera 2.0 Technical Report](https://mlsanity.com/paper/2603.04775/) (2026)
- [Ruyi2.5 Technical Report](https://mlsanity.com/paper/2603.17311/) (2026)
- [EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression](https://mlsanity.com/paper/2609.11058/) (2026)
- [Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference](https://mlsanity.com/paper/2609.37090/) (2026)

## Abstract (arXiv)

With the rapid development of large multimodal models (LMMs), multimodal understanding applications are emerging. As most LMM inference requests originate from edge devices with limited computational capabilities, the predominant inference pipeline involves directly forwarding the input data to an edge server which handles all computations. However, this approach introduces high transmission latency due to limited uplink bandwidth of edge devices and significant computation latency caused by the prohibitive number of visual tokens, thus hindering delay-sensitive tasks and degrading user experience. To address this challenge, we propose a task-oriented feature compression (TOFC) method for multimodal understanding in a device-edge co-inference framework, where visual features are merged by clustering and encoded by a learnable and selective entropy model before feature projection. Specifically, we employ density peaks clustering based on K nearest neighbors to reduce the number of visual features, thereby minimizing both data transmission and computational complexity. Subsequently, a learnable entropy model with hyperprior is utilized to encode and decode merged features, further reducing transmission overhead. To enhance compression efficiency, multiple entropy models are adaptively selected based on the characteristics of the visual features, enabling a more accurate estimation of the probability distribution. Comprehensive experiments on seven visual question answering benchmarks validate the effectiveness of the proposed TOFC method. Results show that TOFC achieves up to 52% reduction in data transmission overhead and 63% reduction in system latency while maintaining identical task performance, compared with neural compression ELIC.