mlsanityData to 13 Oct 2026

CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training

arXiv 2511.12446 ↗First posted 16 Nov 2025cs.CVcited by 1

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Medical visual question answering could support clinical decision making, yet current systems often fail under domain shift and produce answers that are weakly grounded in image evidence. This reliability gap arises when models attend to spurious regions and when retraining or additional labels are impractical at deployment time. We address this setting with CoTBox-TTT, an evidence-first test-time training approach that adapts a vision-language model at inference while keeping all backbones frozen. The method updates only a small set of continuous soft prompts. It identifies question-relevant regions through a visual chain-of-thought signal and encourages answer consistency across the original image and a localized crop. The procedure is label free, and plug and play with diverse backbones. Experiments on medical VQA show that the approach is practical for real deployments. For instance, adding CoTBox-TTT to LLaVA increases closed-ended accuracy by 12.3% on pathVQA.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 37 of its 52 references that are in the map. First posted on arXiv 16 Nov 2025.

Builds on

37 of 52 references.

References · 37

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  2. 2023Segment Anythingcited by 8,195
  3. 2023Visual Instruction Tuningcited by 7,827
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,392
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,296
  12. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 957
25 more
  1. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 687
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 677
  3. 2022Continual Test-Time Domain Adaptationcited by 523
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 472
  5. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 466
  6. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  7. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 446
  8. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 419
  9. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 412
  10. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 400
  11. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 389
  12. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  13. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 334
  14. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 315
  15. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 273
  16. 2022Contrastive Test-Time Adaptationcited by 259
  17. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 140
  18. 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understandingcited by 99
  19. 2022Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clusteringcited by 44
  20. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicinecited by 28
  21. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  22. 2024Hallucination Benchmark in Medical Visual Question Answeringcited by 18
  23. 2025Realistic Test-Time Adaptation of Vision-Language Modelscited by 14
  24. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  25. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
  2. 2025MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models
  3. 2026MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
  4. 2024MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration
  5. 2023Visual Question Answering in the Medical Domain
  6. 2026Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
  7. 2025Benchmarking and Mitigating Sycophancy in Medical Vision Language Models
  8. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs