mlsanityData to 14 Oct 2026

Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models

arXiv 2606.05874 ↗First posted 4 Jun 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Current evaluations for Multimodal Large Language Models (MLLMs) overwhelmingly focus on utility-driven objectives, leaving model behavior under logic-neutral scenarios largely underexplored. Stochasticity is essential in scenarios where multiple actions are equally valid, such as recommending travel itineraries or daily schedules where multiple options have similar utility. In such settings, deterministic policies may lead to repetitive behaviors and reduced coverage of valid alternatives. To bridge this gap, we propose RandomBench, a benchmark designed to evaluate whether MLLMs can maintain distributionally neutral behavior when selecting among equivalent options. We further introduce three metrics, including RI, BCI, BII, to quantify entropy and distributional bias. Experiments reveal a pervasive phenomenon termed Stochastic Collapse, where MLLMs fail to maintain uniform randomness under explicit random instructions, with top-1 probabilities reaching 97% from the ideal one quarter baseline and RI dropping to 0.068 in Claude Sonnet 4.6. Extensive ablation studies further demonstrate that these deviations persist across languages and representation formats, highlighting the robustness of distributional collapse in logic-neutral decision settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 36 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

21 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs4 Nov
  2. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  3. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar

Other references · 18

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  10. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  11. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  12. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 109
6 more
  1. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 40
  2. 2024Language Model Probabilities are Not Calibrated in Numeric Contextscited by 18
  3. 2025Distributionally Robust Graph Out-of-Distribution Recommendation via Diffusion Modelcited by 7
  4. 2026V-FAT: Benchmarking Visual Fidelity Against Text-biascited by 6
  5. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPOcited by 3
  6. 2026The role of System 1 and System 2 semantic memory structure in human and LLM biasescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach
  2. 2026Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
  3. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Models
  4. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models
  5. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
  6. 2026Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models
  7. 2025Unexplored flaws in multiple-choice VQA make benchmarking unreliable
  8. 2026The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models