# LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

arXiv: https://arxiv.org/abs/2408.04957 · HTML: https://mlsanity.com/paper/2408.04957/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 9 Aug 2024.

## Builds on (most cited first, 9 of 9)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [SpatialBot: Precise Spatial Understanding with Vision Language Models](https://mlsanity.com/paper/2406.13642/) (2024)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge](https://mlsanity.com/paper/2311.11860/) (2023)
- [Generating Visual Spatial Description via Holistic 3D Scene Understanding](https://mlsanity.com/paper/2305.11768/) (2023)

## Built on by (most cited first, 7 of 7)

- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Spatial Understanding from Videos: Structured Prompts Meet Simulation Data](https://mlsanity.com/paper/2506.03642/) (2025)
- [E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2502.10455/) (2025)
- [SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video](https://mlsanity.com/paper/2607.01784/) (2026)
- [VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment](https://mlsanity.com/paper/2509.21609/) (2025)

## Abstract (arXiv)

Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images. Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities. In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships. Specifically, the model first constructs a VSD instruction-following dataset using given figure-caption pairs for the three tasks. It then employs LoRA to fine-tune a Large Language and Vision Assistant for VSD, which has 13 billion parameters and supports high-resolution images. Finally, a large language model (Qwen-2) is used to refine the generated sentences, enhancing their diversity and accuracy. LLaVA-VSD demonstrates excellent multimodal conversational capabilities and can follow open-ended instructions to assist with inquiries about object relationships in images.