# Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs

arXiv: https://arxiv.org/abs/2610.05417 · HTML: https://mlsanity.com/paper/2610.05417/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 44 of its 53 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 44)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [PixelLM: Pixel Reasoning with Large Multimodal Model](https://mlsanity.com/paper/2312.02228/) (2023)
- [Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence](https://mlsanity.com/paper/2505.23747/) (2025)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction](https://mlsanity.com/paper/2505.20279/) (2025)
- [From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D](https://mlsanity.com/paper/2503.22976/) (2025)
- [Cambrian-S: Towards Spatial Supersensing in Video](https://mlsanity.com/paper/2511.04670/) (2025)
- [Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors](https://mlsanity.com/paper/2505.24625/) (2025)
- [ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models](https://mlsanity.com/paper/2505.21500/) (2025)
- [OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2506.03135/) (2025)

## Abstract (arXiv)

Recent works augment Vision-Language Models with geometry features from pretrained 3D models, expecting that the geometric signal will boost spatial reasoning. However, we find that simply fusing geometry features and training on standard spatial QA yields only marginal improvements on high-level multi-hop tasks. We attribute this gap to a training-signal problem: standard spatial QA can be largely answered from visual features and language priors, so the geometry pathway receives weak gradients and fails to integrate with the visual features. To provide a training signal that requires geometry, we propose novel-view semantic rendering as an auxiliary training task that requires the model to predict the semantic layout of an unobserved viewpoint, inspired by humans' ability to mentally simulate novel viewpoints during spatial reasoning. This task encourages joint use of both pathways: geometry provides pose-dependent visibility, while vision provides semantic content. Our auxiliary task yields consistent improvements over the geometry-augmented baseline across all three benchmarks (up to +1.6 on VSI-Bench, +2.2 on ReVSI, +2.9 on our 3D-Point-QA dataset) and our full model surpasses prior open-source methods on VSI-Bench and on ReVSI. Project page: https://yuqunw.github.io/Render2Reason/.