# Learning via Self-Consistency for Diffusion-based Video Reasoning

arXiv: https://arxiv.org/abs/2609.36826 · HTML: https://mlsanity.com/paper/2609.36826/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 27 of 27)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark](https://mlsanity.com/paper/2510.26802/) (2025)
- [Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm](https://mlsanity.com/paper/2511.04570/) (2025)
- [Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection](https://mlsanity.com/paper/2412.10891/) (2024)
- [Video-T1: Test-Time Scaling for Video Generation](https://mlsanity.com/paper/2503.18942/) (2025)
- [First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training](https://mlsanity.com/paper/2505.22453/) (2025)
- [Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks](https://mlsanity.com/paper/2511.15065/) (2025)
- [Self-Refining Video Sampling](https://mlsanity.com/paper/2601.18577/) (2026)
- [Demystifying Video Reasoning](https://mlsanity.com/paper/2603.16870/) (2026)
- [Video Generation Models are General-Purpose Vision Learners](https://mlsanity.com/paper/2607.09024/) (2026)
- [What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time](https://mlsanity.com/paper/2603.19880/) (2026)
- [Video Models Reason Early: Exploiting Plan Commitment for Maze Solving](https://mlsanity.com/paper/2603.30043/) (2026)
- [Video Models Can Reason with Verifiable Rewards](https://mlsanity.com/paper/2605.15458/) (2026)
- [Can Generative Video Models Help Pose Estimation?](https://mlsanity.com/paper/2412.16155/) (2024)
- [Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generation](https://mlsanity.com/paper/2605.28230/) (2026)
- [Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning](https://mlsanity.com/paper/2603.27866/) (2026)
- [Toward Early Quality Assessment of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2603.02829/) (2026)

## Abstract (arXiv)

Video generation models have demonstrated emerging zero-shot capabilities for visual reasoning, perception, and other vision tasks. However, diffusion-based video generation is inherently stochastic, while many downstream vision tasks are deterministic. Motivated by the effectiveness of self-consistency in chain-of-thought reasoning for large language models, we investigate whether self-consistency can similarly improve diffusion-based video reasoning. We first introduce a training-free test-time scaling method that samples multiple video generations and aggregates their predictions through self-consistency. Specifically, we aggregate extracted paths, locations, or masks from multiple rollouts into a consensus prediction. To reduce the inference overhead of multi-rollout generation, we read out predictions early in the denoising trajectory, which preserves consensus quality while reducing denoising steps by more than half. We further propose Rejection Fine-Tuning (RFT) to distill consensus predictions into the video generation model. The resulting model internalizes the benefit of multi-sample consensus and requires only a single generation at inference time, while substantially outperforming the original model. Experiments on three tasks, including maze solving, visual search, and referring segmentation, show that both our self-consistency inference and consensus distillation dramatically improve video-based perception and reasoning, without requiring ground-truth videos or task-specific verification. For visual search, self-consistency raises task accuracy from 48.4% for a single generation to 99.0%. The distilled model retains much of the consensus benefit with a single rollout. For 4-by-4 maze solving, consensus-based training improves the single-generation strict success rate from 72.0% to 84.0% with the same inference latency.