mlsanityData to 7 Oct 2026

MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering

arXiv 2303.01239 ↗First posted 2 Mar 2023cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recently, finetuning pretrained Vision-Language Models (VLMs) has been a prevailing paradigm for achieving state-of-the-art performance in Visual Question Answering (VQA). However, as VLMs scale, finetuning full model parameters for a given task in low-resource settings becomes computationally expensive, storage inefficient, and prone to overfitting. Current parameter-efficient tuning methods dramatically reduce the number of tunable parameters, but there still exists a significant performance gap with full finetuning. In this paper, we propose MixPHM, a redundancy-aware parameter-efficient tuning method that outperforms full finetuning in low-resource VQA. Specifically, MixPHM is a lightweight module implemented by multiple PHM-experts in a mixture-of-experts manner. To reduce parameter redundancy, MixPHM reparameterizes expert weights in a low-rank subspace and shares part of the weights inside and across experts. Moreover, based on a quantitative redundancy analysis for adapters, we propose Redundancy Regularization to reduce task-irrelevant redundancy while promoting task-relevant correlation in MixPHM representations. Experiments conducted on VQA v2, GQA, and OK-VQA demonstrate that MixPHM outperforms state-of-the-art parameter-efficient methods and is the only one consistently surpassing full finetuning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 66 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

61 of 66 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering14 Sep
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  5. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  13. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  15. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  18. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  19. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  20. 2021True Few-Shot Learning with Language Modelscited by 343
  21. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  22. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  23. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  24. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  25. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  26. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  27. 2021Hash Layers For Large Sparse Modelscited by 236
  28. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  29. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  30. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  31. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  32. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  33. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  34. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  35. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  36. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  37. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  38. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 67

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 1.

5 papers.

  1. 2024Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuningcited by 20
  2. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Promptscited by 19
  3. 2023ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptationcited by 7
  4. 2024Rethinking Model Redundancy for Low-light Image Enhancementcited by 0
  5. 2026Predictive Reasoning with Augmented Anomaly Contrastive Learning for Compositional Visual Relationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  2. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  3. 2025Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning
  4. 2025CATCH: A Modular Cross-domain Adaptive Template with Hook
  5. 2025AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering
  6. 2025Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  8. 2026VOILA: Value-of-Information Guided Fidelity Selection for Cost-Aware Multimodal Question Answering