# SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes

arXiv: https://arxiv.org/abs/2601.05600 · HTML: https://mlsanity.com/paper/2601.05600/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 23 of 23)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models](https://mlsanity.com/paper/2505.21523/) (2025)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [Grounded Chain-of-Thought for Multimodal Large Language Models](https://mlsanity.com/paper/2503.12799/) (2025)
- [Multimodal Reasoning with Multimodal Knowledge Graph](https://mlsanity.com/paper/2406.02030/) (2024)
- [Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment](https://mlsanity.com/paper/2405.18654/) (2024)
- [Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality](https://mlsanity.com/paper/2305.13812/) (2023)
- [What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning](https://mlsanity.com/paper/2311.01487/) (2023)
- [Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation](https://mlsanity.com/paper/2506.09736/) (2025)
- [VISREAS: Complex Visual Reasoning with Unanswerable Questions](https://mlsanity.com/paper/2403.10534/) (2024)
- [Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning](https://mlsanity.com/paper/2507.13362/) (2025)
- [Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling](https://mlsanity.com/paper/2405.09848/) (2024)
- [QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models](https://mlsanity.com/paper/2405.13014/) (2024)
- [VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?](https://mlsanity.com/paper/2506.11571/) (2025)
- [Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models](https://mlsanity.com/paper/2505.15576/) (2025)
- [From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought](https://mlsanity.com/paper/2507.02984/) (2025)

## Built on by (most cited first, 11 of 11)

- [WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning](https://mlsanity.com/paper/2602.17025/) (2026)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [AMPS: Adaptive Modality Preference Steering via Functional Entropy](https://mlsanity.com/paper/2602.12533/) (2026)
- [MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG](https://mlsanity.com/paper/2604.24564/) (2026)
- [ReGuLaR: Relation-Grounded Latent Reasoning for Large Vision-Language Models](https://mlsanity.com/paper/2605.30587/) (2026)
- [Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG](https://mlsanity.com/paper/2604.27600/) (2026)
- [Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck](https://mlsanity.com/paper/2605.08526/) (2026)
- [MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization](https://mlsanity.com/paper/2605.10784/) (2026)
- [F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking](https://mlsanity.com/paper/2605.12995/) (2026)
- [Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation](https://mlsanity.com/paper/2602.15724/) (2026)
- [RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts](https://mlsanity.com/paper/2607.18470/) (2026)

## Abstract (arXiv)

Multimodal large language models often struggle with faithful reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This reasoning unfaithfulness frequently manifests as hallucinated entities, mis-grounded relations, skipped steps, and over-specified reasoning. Existing preference-based approaches, typically relying on textual perturbations or answer-conditioned rationales, fail to address this challenge as they allow models to exploit language priors to bypass visual grounding. To address this, we propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. By identifying reasoning-critical nodes and perturbing them through four targeted strategies that mimic typical grounding failures, SceneAlign constructs hard negative rationales that remain linguistically plausible but are grounded in inaccurate visual facts. These contrastive pairs are used in Direct Preference Optimization to steer models toward fine-grained, structure-faithful reasoning. Across seven visual reasoning benchmarks, SceneAlign consistently improves answer accuracy and reasoning faithfulness, highlighting the effectiveness of grounding-aware alignment for multimodal reasoning.