mlsanityData to 14 Oct 2026

CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

arXiv 2505.19235 ↗First posted 25 May 2025cs.LG cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision-Language Models (VLMs) excel across diverse tasks but suffer from high inference costs in time and memory. Token sparsity mitigates inefficiencies in token usage, while neuron sparsity reduces high-dimensional computations, both offering promising solutions to enhance efficiency. Recently, these two sparsity paradigms have evolved largely in parallel, fostering the prevailing assumption that they function independently. However, a fundamental yet underexplored question remains: Do they truly operate in isolation, or is there a deeper underlying interplay that has yet to be uncovered? In this paper, we conduct the first comprehensive investigation into this question. By introducing and analyzing the matching mechanism between Core Neurons and Core Tokens, we found that key neurons and tokens for inference mutually influence and reinforce each other. Building on this insight, we propose CoreMatching, a co-adaptive sparse inference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency. Through theoretical analysis and efficiency evaluations, we demonstrate that the proposed method surpasses state-of-the-art baselines on ten image understanding tasks and three hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs reduction and a 10x overall speedup. Code is released at https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 35 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

16 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processing13 Mar
  2. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  3. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  8. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  9. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  10. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  11. 2025HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understandingcited by 19
  12. 2024CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  2. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
  3. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  5. 2025Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signalscited by 13
  6. 2025Focus: A Streaming Concentration Architecture for Efficient Vision-Language Modelscited by 3
  7. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypasscited by 2
  8. 2025Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reportscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking
  2. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
  3. 2026NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning
  4. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
  5. 2024Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
  6. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  7. 2024Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models
  8. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration