mlsanityData to 14 Oct 2026

On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs

arXiv 2610.19892 ↗First posted 14 Oct 2026cs.LG cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning also fits: Visual token pruning and efficiency

Abstract (arXiv)

Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding space.These findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 47 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

40 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Learning to Steer: Input-dependent Steering for Multimodal LLMs18 Aug
  2. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steering6 Jan
  3. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?7 Oct
  4. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  9. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  10. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  11. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  12. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 415
23 more
  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  2. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  3. 2022Contrastive Test-Time Adaptationcited by 261
  4. 2024How to use and interpret activation patchingcited by 214
  5. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  6. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 68
  7. 2025Vision Language Models are Biasedcited by 66
  8. 2024Beyond Model Adaptation at Test Time: A Surveycited by 59
  9. 2023RDumb: A simple approach that questions our progress in continual test-time adaptationcited by 53
  10. 2022ActMAD: Activation Matching to Align Distributions for Test-Time-Trainingcited by 41
  11. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 31
  12. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMscited by 15
  13. 2025LinEAS: End-to-end Learning of Activation Steering with a Distributional Losscited by 14
  14. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Modelscited by 14
  15. 2025Efficient Test-Time Scaling for Small Vision-Language Modelscited by 12
  16. 2025Monitoring Risks in Test-Time Adaptationcited by 11
  17. 2024Test-time Adaptation for Regression by Subspace Alignmentcited by 10
  18. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Modelscited by 8
  19. 2025NEO: No-Optimization Test-Time Adaptation through Latent Re-Centeringcited by 8
  20. 2025Self-Bootstrapping for Versatile Test-Time Adaptationcited by 7
  21. 2025Questioning the Stability of Visual Question Answeringcited by 5
  22. 2025Test-time Correlation Alignmentcited by 4
  23. 2025Are vision language models robust to uncertain inputs?cited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation
  2. 2026Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
  3. 2026Representation Editing for Multimodal Test-Time Adaptation
  4. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  5. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs
  6. 2025Visual Representation Alignment for Multimodal Large Language Models
  7. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Models
  8. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation