# Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MME

arXiv: https://arxiv.org/abs/2606.22862 · HTML: https://mlsanity.com/paper/2606.22862/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 19 of 19)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango](https://mlsanity.com/paper/2209.07686/) (2022)
- [Human Motion Instruction Tuning](https://mlsanity.com/paper/2411.16805/) (2024)
- [FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models](https://mlsanity.com/paper/2405.17814/) (2024)
- [BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models](https://mlsanity.com/paper/2407.15240/) (2024)
- [Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict](https://mlsanity.com/paper/2605.14473/) (2026)
- [SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks](https://mlsanity.com/paper/2605.25492/) (2026)
- [CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Model](https://mlsanity.com/paper/2503.17690/) (2025)
- [Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG](https://mlsanity.com/paper/2605.28044/) (2026)
- [Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit](https://mlsanity.com/paper/2605.28873/) (2026)
- [Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries](https://mlsanity.com/paper/2605.18891/) (2026)
- [BiasIG: Benchmarking Multi-dimensional Social Biases in Text-to-Image Models](https://mlsanity.com/paper/2604.11934/) (2026)
- [MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.24253/) (2025)

## Built on by (most cited first, 1 of 1)

- [AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning](https://mlsanity.com/paper/2608.29988/) (2026)

## Abstract (arXiv)

Forced chain-of-thought (CoT) is widely assumed to make vision-language models more reliable on video question answering. We propose a small three-probe evaluation recipe to test that assumption: paired accuracy across direct, CoT, answer-first, and no-video conditions; a counterfactual video-swap diagnostic over the CoT chains; and a four-rung visual-degradation ladder. Each probe is reported under both a strict and a permissive regex scorer, with multiplicity correction over a manuscript-declared primary family. Applied to Qwen2.5-VL on Video-MME subsets, the recipe returns a two-part finding. The CoT chains are strongly video-conditioned: swapping the input video collapses chain overlap and flips most final letters, the opposite of what a "boilerplate-chain" null would predict. Yet on the same data, forced CoT does not improve MCQ accuracy, and on the smaller 7B model it produces a small but statistically supported drop under a post-hoc primary scorer choice. We do not claim this generalizes beyond the Qwen2.5-VL / Video-MME instantiation; the raw responses and a single recomputation script will be released with the supplementary material so every number can be re-derived.