# Galaxy Walker: Geometry-aware VLMs For Galaxy-scale Understanding

arXiv: https://arxiv.org/abs/2503.18578 · HTML: https://mlsanity.com/paper/2503.18578/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxies](https://mlsanity.com/paper/2102.08414/) (2021)
- [SphereSR: 360deg Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representation](https://mlsanity.com/paper/2112.06536/) (2021)
- [GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation](https://mlsanity.com/paper/2406.11503/) (2024)
- [GeoCode: Interpretable Shape Programs](https://mlsanity.com/paper/2212.11715/) (2022)
- [Self-supervised similarity search for large scientific datasets](https://mlsanity.com/paper/2110.13151/) (2021)
- [DreamNet: A Deep Riemannian Network based on SPD Manifold Learning for Visual Classification](https://mlsanity.com/paper/2206.07967/) (2022)

## Built on by (most cited first, 4 of 4)

- [Hyperbolic Busemann Neural Networks](https://mlsanity.com/paper/2602.18858/) (2026)
- [AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models](https://mlsanity.com/paper/2509.12715/) (2025)
- [Toward a Unified Geometry Understanding: Riemannian Diffusion Framework for Graph Generation and Prediction](https://mlsanity.com/paper/2510.04522/) (2025)
- [Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models](https://mlsanity.com/paper/2606.00275/) (2026)

## Abstract (arXiv)

Modern vision-language models (VLMs) develop patch embedding and convolution backbone within vector space, especially Euclidean ones, at the very founding. When expanding VLMs to a galaxy scale for understanding astronomical phenomena, the integration of spherical space for planetary orbits and hyperbolic spaces for black holes raises two formidable challenges. a) The current pre-training model is confined to Euclidean space rather than a comprehensive geometric embedding. b) The predominant architecture lacks suitable backbones for anisotropic physical geometries. In this paper, we introduced Galaxy-Walker, a geometry-aware VLM, for the universe-level vision understanding tasks. We proposed the geometry prompt that generates geometry tokens by random walks across diverse spaces on a multi-scale physical graph, along with a geometry adapter that compresses and reshapes the space anisotropy in a mixture-of-experts manner. Extensive experiments demonstrate the effectiveness of our approach, with Galaxy-Walker achieving state-of-the-art performance in both galaxy property estimation (R2 scores up to 0.91) and morphology classification tasks (up to +0.17 F1 improvement in challenging features), significantly outperforming both domain-specific models and general-purpose VLMs.