mlsanityData to 7 Oct 2026

Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs

arXiv 2404.07449 ↗First posted 11 Apr 2024cs.CVcited by 48

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA). However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness. Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location. In this work, we explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs. We discover optimal coordinate representations, data-efficient instruction fine-tuning objectives, and pseudo-data generation strategies that lead to improved spatial awareness in V-LLMs. Additionally, our resulting model improves VQA across image and video domains, reduces undesired hallucination, and generates better contextual object descriptions. Experiments across 5 vision-language tasks involving 14 different datasets establish the clear performance improvements achieved by our proposed framework.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 63 of its 77 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

63 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2015Fast R-CNNcited by 2,930
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  15. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  18. 2022Language-driven Semantic Segmentationcited by 711
  19. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  20. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  21. 2021Extract Free Dense Labels from CLIPcited by 580
  22. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  23. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  24. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  25. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  26. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  27. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  28. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  29. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  30. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  31. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  32. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  33. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  34. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  35. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  36. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  37. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  38. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 24; 2026: 13.

48 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  2. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  3. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  4. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  5. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  6. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  7. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  8. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  9. 2024Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Modelscited by 17
  10. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  11. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understandingcited by 16
  12. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
36 more
  1. 2026SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLMcited by 11
  2. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  3. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  4. 2025PRS-Med: Position Reasoning Segmentation in Medical Imagingcited by 8
  5. 2025Text-Aware Image Restoration with Diffusion Modelscited by 5
  6. 2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
  7. 2024Evaluating and Enhancing Trustworthiness of LLMs in Perception Taskscited by 4
  8. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervisioncited by 4
  9. 2025Probing LLM World Models: Enhancing Guesstimation with Wisdom of Crowds Decodingcited by 4
  10. 2025Ground-V: Teaching VLMs to Ground Complex Instructions in Pixelscited by 4
  11. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?cited by 3
  12. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  13. 2025Spatial Preference Rewarding for MLLMs Spatial Understandingcited by 3
  14. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 2
  15. 2024CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Modelscited by 2
  16. 2025VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Modelscited by 2
  17. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  18. 2025VIKSER: Visual Knowledge-Driven Self-Reinforcing Reasoning Frameworkcited by 1
  19. 2025Compositional Image-Text Matching and Retrieval by Grounding Entitiescited by 1
  20. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMscited by 1
  21. 2025CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentationcited by 1
  22. 2026IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidancecited by 1
  23. 2026From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Designcited by 1
  24. 2024Navigate Complex Physical Worlds via Geometrically Constrained LLMcited by 0
  25. 2025Experimental Evaluation of Static Image Sub-Region-Based Search Models Using CLIPcited by 0
  26. 2025"Does the cafe entrance look accessible? Where is the door?" Towards Geospatial AI Agents for Visual Inquiriescited by 0
  27. 2025Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understandingcited by 0
  28. 2026Reasoning with Autoregressive-Diffusion Collaborative Thoughtscited by 0
  29. 2026Test-Time Computing for Referring Multimodal Large Language Modelscited by 0
  30. 2026Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoderscited by 0
  31. 2026Leveraging Weighted Syntactic and Semantic Context Assessment Summary (wSSAS) Towards Text Categorization Using LLMscited by 0
  32. 2026Consistency Analysis of Sentiment Predictions using Syntactic & Semantic Context Assessment Summarization (SSAS)cited by 0
  33. 2026SpatialForge: Bootstrapping 3D-Aware Spatial Reasoning from Open-World 2D Imagescited by 0
  34. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  35. 2026Gaze-to-text Generation: Beyond Categorical Decoding of Human Attentioncited by 0
  36. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models
  4. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
  5. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
  6. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
  7. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  8. 2025Enhancing Spatial Reasoning through Visual and Textual Thinking