mlsanityData to 14 Oct 2026

SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation

arXiv 2502.13143 ↗First posted 18 Feb 2025cs.RO cs.AI cs.CVcited by 46

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation-a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the "plug-in" direction of a USB or the "handle" direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7% successful rate on Open6DOR and zero-shot 74.9% successful rate on SIMPLER-Env.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 139 of its 181 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

139 of 181 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation3 Sep
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  4. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors18 Mar
  5. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning18 Mar
  6. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  8. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 130

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2023Segment Anythingcited by 8,234
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  13. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  14. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  15. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  16. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  17. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  18. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  19. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  20. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  21. 2020Point Transformercited by 1,470
  22. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  23. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  24. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  25. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  26. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  27. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  28. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  29. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  30. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  31. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  32. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  33. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  34. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  35. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  36. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  37. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  38. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511

50 most cited of 130 All 130 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 28; 2026: 18.

46 papers.

Sort:

By area:

  1. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  2. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  3. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  4. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  5. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  6. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
  7. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 43
  8. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  9. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  10. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 29
  11. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  12. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
34 more
  1. 2026Orient Anything V2: Unifying Orientation and Rotation Understandingcited by 23
  2. 2025DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulationcited by 22
  3. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 20
  4. 2025Reasoning in Space via Grounding in the Worldcited by 19
  5. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  6. 2025Conditioning Matters: Training Diffusion Policies is Faster Than You Thinkcited by 9
  7. 2025ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Taskscited by 9
  8. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Modelscited by 9
  9. 2025Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimationcited by 6
  10. 2025Physically Ground Commonsense Knowledge for Articulated Object Manipulation with Analytic Conceptscited by 5
  11. 2025NoTVLA: Semantics-Preserving Robot Adaptation via Narrative Action Interfacescited by 4
  12. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  13. 2026Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agentscited by 4
  14. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
  15. 2025MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learningcited by 3
  16. 2025Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulationcited by 3
  17. 2026ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Modelscited by 3
  18. 2026MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalizationcited by 3
  19. 2026Emerging Extrinsic Dexterity in Cluttered Scenes via Dynamics-aware Policy Learningcited by 2
  20. 2026From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulationcited by 2
  21. 2025T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Modelscited by 1
  22. 2025Obstruction reasoning for robotic graspingcited by 1
  23. 2025Bridging Scale Discrepancies in Robotic Control via Language-Based Action Representationscited by 1
  24. 2026HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Graspingcited by 1
  25. 2026Action with Visual Primitivescited by 1
  26. 2026ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?cited by 1
  27. 2026GuideFetch: A Task Coordination Framework for Concurrent Navigation and Object Retrieval in Assistive Robot Dogscited by 1
  28. 2025FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Modelscited by 0
  29. 2026AnchorVLA4D: an Anchor-Based Spatial-Temporal Vision-Language-Action Model for Robotic Manipulationcited by 0
  30. 2026A Cross-view Fusion Framework for Robust 6-DoF Grasp Pose Estimationcited by 0
  31. 2026Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Modelscited by 0
  32. 2026Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluationcited by 0
  33. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0
  34. 2026Soft Spatial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  2. 2025PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation
  3. 2023Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning
  4. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  5. 2024Task-oriented Robotic Manipulation with Vision Language Models
  6. 2025AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoning
  7. 2023A Universal Semantic-Geometric Representation for Robotic Manipulation
  8. 2026VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation