Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement
Not on the map. Why?
Abstract (arXiv)
Interleaved thinking, where a unified multimodal model alternates between textual reasoning and visual generation, has shown promise on spatial and physical tasks. However, in complex long-chain scenarios, we identify a fundamental failure mode: generated images diverge from the textual context while subsequent text ignores the visual evidence, causing the two modalities to alternate without genuinely informing each other. We term this Modal Isolation and attribute it to compounding information loss at modality boundaries. We decompose each reasoning cycle into atomic operations and define modality transition loss, quantifying cross-modal hallucination (text-to-image) and visual utilization deficit (image-to-text) at each boundary. We propose MoTiF (Modality Tiransition Fidelity), a two-stage training framework that directly optimizes these transitions: Reflective SFT trains the model to detect and recover from erroneous visual outputs; Flow-GRPO improves image generation fidelity via reinforcement learning. All training signals in MoTiF derive from transition-level fidelity rather than end-task accuracy. Across four visual puzzle benchmarks, this transition-level supervision substantially improves both cross-modal coherence and final task accuracy. The results demonstrate that effective interleaved reasoning requires explicit structural supervision at modality boundaries, not merely scaling or end-task optimization.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 11 Jun 2026.
Builds on
24 of 31 references.
References · 24
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
- 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
- 2023Generating Images with Multimodal Language Modelscited by 260
- 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
- 2025GRIT: Teaching MLLMs to Think with Imagescited by 125
12 more
- 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
- 2025Thinking with Generated Imagescited by 73
- 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 56
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
- 2024Interleaved-Modal Chain-of-Thoughtcited by 51
- 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 49
- 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
- 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoningcited by 32
- 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thoughtcited by 21
- 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
- 2026The Trinity of Consistency as a Defining Principle for General World Modelscited by 6
- 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimizationcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
- 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
- 2025On the Faithfulness of Visual Thinking: Measurement and Enhancement
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
- 2026Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
- 2026R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning
- 2026Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
- 2026Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models