# Space-LLaVA: a Vision-Language Model Adapted to Extraterrestrial Applications

arXiv: https://arxiv.org/abs/2408.05924 · HTML: https://mlsanity.com/paper/2408.05924/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 69 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Language Models are Spacecraft Operators](https://mlsanity.com/paper/2404.00413/) (2024)

## Built on by (most cited first, 14 of 14)

- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2510.07181/) (2025)
- [Visual Language Models as Operator Agents in the Space Domain](https://mlsanity.com/paper/2501.07802/) (2025)
- [Language-Conditioned Safe Trajectory Generation for Spacecraft Rendezvous](https://mlsanity.com/paper/2512.09111/) (2025)
- [OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks](https://mlsanity.com/paper/2604.08539/) (2026)
- [Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning](https://mlsanity.com/paper/2603.14811/) (2026)
- [LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration](https://mlsanity.com/paper/2603.24696/) (2026)
- [RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation](https://mlsanity.com/paper/2603.27112/) (2026)
- [Intent-aligned Autonomous Spacecraft Guidance via Reasoning Models](https://mlsanity.com/paper/2604.17176/) (2026)
- [Semantic Constraint Synthesis for Adaptive Trajectory Optimization via Large Language Models](https://mlsanity.com/paper/2606.04123/) (2026)
- [A Systematic Evaluation of Retrieval-Augmented Generation and Language Models for Space Operations](https://mlsanity.com/paper/2605.27444/) (2026)
- [Exploring LLM Capabilities for Situational Understanding and COLREG compliance on real-world maritime navigation scenarios](https://mlsanity.com/paper/2608.08281/) (2026)
- [Question-Specific Knowledge Graphs for Efficient Visual Reasoning](https://mlsanity.com/paper/2609.35942/) (2026)
- [OrbitTAMP: Grounding Language Models for Task and Motion Planning in Spacecraft Rendezvous](https://mlsanity.com/paper/2610.01093/) (2026)

## Abstract (arXiv)

Foundation Models (FMs), e.g., large language models, possess attributes of intelligence which offer promise to endow a robot with the contextual understanding necessary to navigate complex, unstructured tasks in the wild. We see three core challenges in the future of space robotics that motivate building an FM for the space robotics community: 1) Scalability of ground-in-the-loop operations; 2) Generalizing prior knowledge to novel environments; and 3) Multi-modality in tasks and sensor data. As a first-step towards a space foundation model, we programmatically augment three extraterrestrial databases with fine-grained language annotations inspired by the sensory reasoning necessary to e.g., identify a site of scientific interest on Mars, building a synthetic dataset of visual-question-answer and visual instruction-following tuples. We fine-tune a pre-trained LLaVA 13B checkpoint on our augmented dataset to adapt a Vision-Language Model (VLM) to the visual semantic features in an extraterrestrial environment, demonstrating FMs as a tool for specialization and enhancing a VLM's zero-shot performance on unseen task types in comparison to state-of-the-art VLMs. Ablation studies show that fine-tuning the language backbone and vision-language adapter in concert is key to facilitate adaption while a small percentage, e.g., 20%, of the pre-training data can be used to safeguard against catastrophic forgetting.