mlsanityData to 7 Oct 2026

ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

arXiv 2603.26486 ↗First posted 27 Mar 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 61 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

48 of 61 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs31 Jan
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  7. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models2 Nov

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
29 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2016Adversarial examples in the physical worldcited by 1,762
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2022Continual Test-Time Domain Adaptationcited by 520
  10. 2018nocaps: novel object captioning at scalecited by 444
  11. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  13. 2022Contrastive Test-Time Adaptationcited by 258
  14. 2022Test-Time Training with Masked Autoencoderscited by 208
  15. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  16. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  17. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  18. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  19. 2017A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortionscited by 107
  20. 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
  21. 2016Measuring Neural Net Robustness with Constraintscited by 90
  22. 2021MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaptioncited by 65
  23. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  24. 2017Comparing deep neural networks against humans: object recognition when the signal gets weakercited by 34
  25. 2017CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognitioncited by 32
  26. 2023Convolutional Visual Prompt for Robust Visual Perceptioncited by 17
  27. 2025Test-Time Visual In-Context Tuningcited by 7
  28. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 7
  29. 2025Improving Alignment in LVLMs with Debiased Self-Judgmentcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMscited by 0
  2. 2026When Should the Count Change? Learning State Maintenance for Causal Video Countingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  2. 2025Mitigating Image Captioning Hallucinations in Vision-Language Models
  3. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models
  4. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  5. 2025Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScore
  6. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
  7. 2026Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation
  8. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering