mlsanityData to 7 Oct 2026

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

arXiv 2609.29278 ↗First posted 24 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought (CoT) instructions can distort multiple-choice VLM evaluation when a scorer appends a reasoning cue but reads answer-label logits before the model generates any rationale. We call this CoT-prefix scoring. On ScienceQA, Qwen2.5-VL-7B drops from 80.76% to 45.48%, and across five option-content permutations 93.54% of CoT-prefix predictions select the first slot. Condition-matched linear probes recover 78.94% from the same hidden states, while free generation restores 75.24%, showing that the answer often survives the prefix and the immediate readout fails. Vocabulary and layer diagnostics explain the mismatch: probability mass moves toward continuation tokens, while answer information remains linearly accessible in late layers. The effect recurs with varying severity across datasets and models, though not universally. These results show that CoT-prefix scoring can confound model knowledge with an evaluation-interface mismatch and should be avoided unless the requested and scored output events are aligned.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 21 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

18 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 17

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  10. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  11. 2016A Diagram Is Worth A Dozen Imagescited by 793
  12. 2019Designing and Interpreting Probes with Control Taskscited by 608
5 more
  1. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  2. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  3. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  4. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  5. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Step
  2. 2026Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
  3. 2023Measuring Faithfulness in Chain-of-Thought Reasoning
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  5. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding
  6. 2024Improve Vision Language Model Chain-of-thought Reasoning
  7. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  8. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning