# GeoExplain: Multimodal Reasoning based on Hierarchy of Visual Information in Street View

arXiv: https://arxiv.org/abs/2506.16633 · HTML: https://mlsanity.com/paper/2506.16633/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 85 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling](https://mlsanity.com/paper/2108.01453/) (2021)
- [Learning Generalized Zero-Shot Learners for Open-Domain Image Geolocalization](https://mlsanity.com/paper/2302.00275/) (2023)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [Maria: A Visual Experience Powered Conversational Agent](https://mlsanity.com/paper/2105.13073/) (2021)
- [Image-Based Geolocation Using Large Vision-Language Models](https://mlsanity.com/paper/2408.09474/) (2024)
- [VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks](https://mlsanity.com/paper/2104.07921/) (2021)
- [Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework](https://mlsanity.com/paper/2408.11312/) (2024)
- [Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game](https://mlsanity.com/paper/2406.11012/) (2024)
- [Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding](https://mlsanity.com/paper/2406.18925/) (2024)
- [Efficient Attention Mechanism for Visual Dialog that can Handle All the Interactions between Multiple Inputs](https://mlsanity.com/paper/1911.11390/) (2019)

## Built on by (most cited first, 3 of 3)

- [GeoRC: A Benchmark for Geolocation Reasoning Chains](https://mlsanity.com/paper/2601.21278/) (2026)
- [HoloGeo: Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning](https://mlsanity.com/paper/2607.15255/) (2026)
- [Perceive to Hypothesize, Verify to Ground: An Agentic Reasoning Framework for Open-World Geo-Localization](https://mlsanity.com/paper/2608.29880/) (2026)

## Abstract (arXiv)

Multimodal reasoning is a process of understanding, integrating and inferring information across different data modalities. It has recently attracted surging academic attention. Although there are various tasks for evaluating multimodal reasoning ability, they still have limitations. Reasoning on hierarchical visual clues at different levels of granularity, i.e., local details and global context, is of little discussion, despite its frequent involvement in human reasoning. To bridge the gap, we introduce a challenging dataset, namely GeoExplain, which evaluates explainable geo-localization. Given a street view image, the task is to predict its location and provide a detailed explanation. GeoExplain consists of 40350 panoramas-location-explanation tuples. Each instance contains a set of street-view panoramas, a location on street level, and human-expert explanations describing how the location can be inferred from the visual content of panoramas. Additionally, we present a multimodal and multilevel reasoning method, namely SightSense which can make predictions and generate a comprehensive explanation. Our analysis and experiments demonstrate its outstanding performance in GeoExplain.