mlsanityData to 14 Oct 2026

CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models

arXiv 2608.04509 ↗First posted 5 Aug 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer. Reliable models must identify the trustworthy source and abstain when neither is adequate. Existing post-training objectives score instances independently and therefore do not enforce coherent behavior under counterfactual evidence changes. We introduce CARGO-VL, a group-relative framework that optimizes matched variants covering aligned, image-correct, text-correct, and both-wrong (A/V/T/N) evidence states as one bundle. Its objective couples condition-wise correctness with transition rewards for answer invariance, source equivariance, and answer-to-abstention switching, while a primal-dual controller balances unsafe answers against excessive deferral. We also contribute XMC (eXtended Modal Conflict), a four-condition conflict training resource, and evaluate transfer on CMC-Bench and Modality-Bias. Across multiple seeds, CARGO-VL improves conflict handling, unsupported-answer avoidance, and modality balance over pointwise baselines. Ablations identify complementary benefits from relational transition signals and adaptive risk control, supporting counterfactual consistency as a practical objective for reliable multimodal evidence arbitration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 16 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

12 of 16 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict11 Apr

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2017Selective Classification for Deep Neural Networkscited by 852
  11. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evidence-RL: Towards Evidence-intensive Visual Reasoning
  2. 2026FPCO-Dialog: A Multi-Turn False-Premise Benchmark for Correction and Cooperation in Vision-Language Models
  3. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
  4. 2026Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
  5. 2026Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems
  6. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  7. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  8. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models