# Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method

arXiv: https://arxiv.org/abs/2505.14197 · HTML: https://mlsanity.com/paper/2505.14197/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 86 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 16 of 16)

- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11051/) (2025)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [Learning High-Quality Navigation and Zooming on Omnidirectional Images in Virtual Reality](https://mlsanity.com/paper/2405.00351/) (2024)

## Built on by (most cited first, 13 of 13)

- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026)
- [Omnidirectional Spatial Modeling from Correlated Panoramas](https://mlsanity.com/paper/2509.02164/) (2025)
- [PanoWorld: Towards Spatial Supersensing in 360^circ Panorama World](https://mlsanity.com/paper/2605.13169/) (2026)
- [360deg Image Perception with MLLMs: A Comprehensive Benchmark and a Training-Free Method](https://mlsanity.com/paper/2603.16179/) (2026)
- [FireScope: Wildfire Risk Raster Prediction with a Chain-of-Thought Oracle](https://mlsanity.com/paper/2511.17171/) (2025)
- [What if? Emulative Simulation with World Models for Situated Reasoning](https://mlsanity.com/paper/2603.06445/) (2026)
- [Panoramic Affordance Prediction](https://mlsanity.com/paper/2603.15558/) (2026)
- [Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling](https://mlsanity.com/paper/2606.27745/) (2026)
- [EAGLE-360: Embodied Active Global-to-Local Exploration in 360^circ](https://mlsanity.com/paper/2607.02479/) (2026)
- [Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images](https://mlsanity.com/paper/2605.12413/) (2026)
- [OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning](https://mlsanity.com/paper/2606.30378/) (2026)

## Abstract (arXiv)

Omnidirectional images (ODIs), with their 360deg field of view, provide unparalleled spatial awareness for immersive applications like augmented reality and embodied AI. However, the capability of existing multi-modal large language models (MLLMs) to comprehend and reason about such panoramic scenes remains underexplored. This paper addresses this gap by introducing OmniVQA, the first dataset and conducting the first benchmark for omnidirectional visual question answering. Our evaluation of state-of-the-art MLLMs reveals significant limitations in handling omnidirectional visual question answering, highlighting persistent challenges in object localization, feature extraction, and hallucination suppression within panoramic contexts. These results underscore the disconnect between current MLLM capabilities and the demands of omnidirectional visual understanding, which calls for dedicated architectural or training innovations tailored to 360deg imagery. Building on the OmniVQA dataset and benchmark, we further introduce a rule-based reinforcement learning method, 360-R1, based on Qwen2.5-VL-Instruct. Concretely, we modify the group relative policy optimization (GRPO) by proposing three novel reward functions: (1) reasoning process similarity reward, (2) answer semantic accuracy reward, and (3) structured format compliance reward. Extensive experiments on our OmniVQA demonstrate the superiority of our proposed method in omnidirectional space (+6% improvement).