mlsanityData to 7 Oct 2026

FORUM: Frozen Outputs Reconciled Using Model Agreement for Visual Grounding

arXiv 2609.37488 ↗First posted 27 Sep 2026cs.CV cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Frozen multimodal large language models (MLLMs) now solve standard referring expression comprehension with a single prompted call, yet on adversarial benchmarks with same-category distractors and negation, even the largest models are confidently wrong, and resampling repeats the error. Models built from different data and architectures rarely fall for the same confounder, so their agreement is a strong label-free signal of the correct target. We present FORUM, a training-free test-time fusion of frozen MLLMs guided by two fixed geometric rules: agreement-based selection keeps the region supported by the most distinct models, and medoid localization returns an actual member box instead of a coordinate average, so one loose prediction cannot shift the answer. Fusing three open MLLMs, FORUM surpasses the 397B-parameter published reference by a relative 5% in mean accuracy on the adversarial Ref-Adv-s benchmark, and a plain averaging ensemble by 15%. The gains transfer to standard RefCOCO+, and a balanced lineup with no dominant member still surpasses the 397B model by 5%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 39 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

31 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision27 Jan
  2. 2025Qwen3-VL Technical Report26 Nov
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun

Other references · 28

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2016Modeling Context in Referring Expressionscited by 1,054
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
16 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  3. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  4. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  5. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  6. 2024More Agents Is All You Needcited by 174
  7. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  8. 2020Better Aggregation in Test-Time Augmentationcited by 121
  9. 2020Referring Expression Comprehension: A Survey of Methods and Datasetscited by 78
  10. 2024Toward Visual Grounding: A Surveycited by 52
  11. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 23
  12. 2025DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergycited by 13
  13. 2026Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Taskscited by 8
  14. 2025DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Modelcited by 3
  15. 2026See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflectioncited by 3
  16. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities
  2. 2025Test-Time Warmup for Multimodal Large Language Models
  3. 2026PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs
  4. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Models
  5. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  6. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models