mlsanityData to 14 Oct 2026

Shifting AI Efficiency From Model-Centric to Data-Centric Compression

arXiv 2505.19147 ↗First posted 25 May 2025cs.CL cs.AI cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

The advancement of large language models (LLMs) and multi-modal LLMs (MLLMs) has historically relied on scaling model parameters. However, as hardware limits constrain further model growth, the primary computational bottleneck has shifted to the quadratic cost of self-attention over increasingly long sequences by ultra-long text contexts, high-resolution images, and extended videos. In this position paper, we argue that the focus of research for efficient artificial intelligence (AI) is shifting from model-centric compression to data-centric compression. We position data-centric compression as the emerging paradigm, which improves AI efficiency by directly compressing the volume of data processed during model training or inference. To formalize this shift, we establish a unified framework for existing efficiency strategies and demonstrate why it constitutes a crucial paradigm change for long-context AI. We then systematically review the landscape of data-centric compression methods, analyzing their benefits across diverse scenarios. Finally, we outline key challenges and promising future research directions. Our work aims to provide a novel perspective on AI efficiency, synthesize existing efforts, and catalyze innovation to address the challenges posed by ever-increasing context lengths.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 146 of its 184 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

146 of 184 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference24 Feb
  2. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?17 Feb
  3. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More17 Feb
  4. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models9 Jan
  5. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph4 Jan
  6. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs8 Dec
  7. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  8. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs4 Dec
  9. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model16 Nov
  10. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  11. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  12. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
6 more
  1. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  4. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Models12 Dec
  5. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Models27 May
  6. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers27 May

Other references · 128

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  11. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  12. 2025Qwen3 Technical Reportcited by 8,820
38 more
  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2023Let's Verify Step by Stepcited by 3,789
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  13. 2023Qwen Technical Reportcited by 3,344
  14. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  15. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  16. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  17. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  18. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  19. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  20. 2023A Survey of Large Language Modelscited by 2,576
  21. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  22. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  23. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  24. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  25. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  26. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  27. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  28. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  29. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  30. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  31. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  32. 2020Knowledge Distillation: A Surveycited by 1,603
  33. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  34. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  35. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  36. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  37. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  38. 2016Pruning Filters for Efficient ConvNetscited by 1,143

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 7.

29 papers.

Sort:

By area:

  1. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 93
  2. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 63
  3. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  4. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  5. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 27
  6. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  7. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  8. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 22
  9. 2025AI for Service: Proactive Assistance with AI Glassescited by 20
  10. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 17
  11. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 16
  12. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
17 more
  1. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 14
  2. 2025PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Eracited by 12
  3. 2026Innovator-VL: A Multimodal Large Language Model for Scientific Discoverycited by 8
  4. 2025IPCV: Information-Preserving Compression for MLLM Visual Encoderscited by 5
  5. 2026EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistantcited by 5
  6. 2026SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaborationcited by 4
  7. 2025ARC-Encoder: learning compressed text representations for large language modelscited by 3
  8. 2025VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstructioncited by 3
  9. 2025Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMscited by 2
  10. 2025OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learningcited by 2
  11. 2026V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Modelscited by 2
  12. 2026LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMscited by 2
  13. 2025Differentially-private text generation degrades output language qualitycited by 1
  14. 2026Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigationcited by 1
  15. 2026HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Modelscited by 1
  16. 2025Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generationcited by 0
  17. 2025UNSEEN: Enhancing Dataset Pruning from a Generalization Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
  2. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  3. 2026Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model
  4. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
  5. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks
  6. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
  7. 2023A Survey on Model Compression for Large Language Models
  8. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression