mlsanityData to 14 Oct 2026

Test-Time Warmup for Multimodal Large Language Models

arXiv 2509.10641 ↗First posted 12 Sep 2025cs.LG cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) hold great promise for advanced reasoning at the intersection of text and images, yet they have not fully realized this potential. MLLMs typically integrate an LLM, a vision encoder, and a connector that maps the vision encoder's embeddings into the LLM's text embedding space. Although each component is pretrained on massive datasets with billions of samples, the entire multimodal model is typically trained on only thousands (or a few million) samples, which can result in weak performance on complex reasoning tasks. To address these shortcomings, instead of relying on extensive labeled datasets for fine-tuning, we propose a Test-Time Warmup method that adapts the MLLM per test instance by leveraging data from weakly supervised auxiliary tasks. With our approach, we observe a relative performance improvement of 4.03% on MMMU, 5.28% on VQA-Rad, and 1.63% on GQA on the Llama-Vision-Instruct model. Our method demonstrates that 'warming up' before inference can enhance MLLMs' robustness across diverse reasoning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 41 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

25 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation7 Jan
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  9. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  10. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  11. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  12. 2019Unsupervised Domain Adaptation through Self-Supervisioncited by 116
11 more
  1. 2019Test-Time Training with Self-Supervision for Generalization under Distribution Shiftscited by 59
  2. 2018Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Trainingcited by 31
  3. 2020Auxiliary Task Reweighting for Minimum-data Learningcited by 23
  4. 2023SPA: A Graph Spectral Alignment Perspective for Domain Adaptationcited by 22
  5. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  6. 2018Unsupervised Domain Adaptation with Adversarial Residual Transform Networkscited by 14
  7. 2024Neural networks for abstraction and reasoning: Towards broad generalization in machinescited by 13
  8. 2025Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstractioncited by 12
  9. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
  10. 2024Free Form Medical Visual Question Answering in Radiologycited by 4
  11. 2022Auxiliary task discovery through generate-and-testcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  2. 2026Test-Time Computing for Referring Multimodal Large Language Models
  3. 2025Learning to Inference Adaptively for Multimodal Large Language Models
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  5. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
  6. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  7. 2026MoR-MLLM: Mixture of Recursions for Efficient Multimodal Large Language Models
  8. 2024Efficient Multimodal Large Language Models: A Survey