mlsanityData to 14 Oct 2026

UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers

arXiv 2301.13741 ↗First posted 31 Jan 2023cs.CV cs.CL cs.LGcited by 54

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Real-world data contains a vast amount of multimodal information, among which vision and language are the two most representative modalities. Moreover, increasingly heavier models, textite.textitg., Transformers, have attracted the attention of researchers to model compression. However, how to compress multimodal models, especially vison-language Transformers, is still under-explored. This paper proposes the Unified and Progressive Pruning (textbfUPop) as a universal vison-language Transformer compression framework, which incorporates 1) unifiedly searching multimodal subnets in a continuous optimization space from the original model, which enables automatic assignment of pruning ratios among compressible modalities and structures; 2) progressively searching and retraining the subnet, which maintains convergence between the search and retrain to attain higher compression ratios. Experiments on various tasks, datasets, and model architectures demonstrate the effectiveness and versatility of the proposed UPop framework. The code is available at https://github.com/sdc17/UPop.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 67 of its 84 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

67 of 84 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
38 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  13. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  14. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  16. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  17. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  18. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  19. 2019Are Sixteen Heads Really Better than One?cited by 816
  20. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  21. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  22. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  23. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  24. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  25. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  28. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  29. 2022DeiT III: Revenge of the ViTcited by 419
  30. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  31. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  32. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  33. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  34. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  35. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  36. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  37. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  38. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 19; 2025: 13; 2026: 11.

54 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  3. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 111
  4. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  6. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  7. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  8. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 43
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  11. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  12. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 30
38 more
  1. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  2. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  3. 2024Isomorphic Pruning for Vision Modelscited by 21
  4. 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisitioncited by 21
  5. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Trainingcited by 20
  6. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  7. 2024The Need for Speed: Pruning Transformers with One Recipecited by 16
  8. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  9. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  10. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  11. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Modelscited by 11
  12. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 11
  13. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  14. 2023SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Modelscited by 7
  15. 2024Skipping Computations in Multimodal LLMscited by 7
  16. 2025CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimizationcited by 7
  17. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layerscited by 7
  18. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  19. 2023Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Modelscited by 5
  20. 2024Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunablecited by 4
  21. 2023Uncovering the Hidden Cost of Model Compressioncited by 3
  22. 2024Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Headscited by 3
  23. 2025Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLMcited by 3
  24. 2025Adaptive Pruning of Pretrained Transformer via Differential Inclusionscited by 2
  25. 2023Bootstrapping SparseFormers from Vision Foundation Modelscited by 1
  26. 2024NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionalitycited by 1
  27. 2025Safe Vision-Language Models via Unsafe Weights Manipulationcited by 1
  28. 2025CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Modelscited by 1
  29. 2026ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruningcited by 1
  30. 2026Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Modelscited by 1
  31. 2024Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Modelscited by 0
  32. 2024AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Maskingcited by 0
  33. 2025EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correctioncited by 0
  34. 2025Detection Transformers Under the Knife: A Neuroscience-Inspired Approach to Ablationscited by 0
  35. 2026CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compressioncited by 0
  36. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0
  37. 2026Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understandingcited by 0
  38. 2026When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compressioncited by 0

50 most cited of 54 All 54 (JSON)