mlsanityData to 14 Oct 2026

RoboGolf: Mastering Real-World Minigolf with a Reflective Multi-Modality Vision-Language Model

arXiv 2406.10157 ↗First posted 14 Jun 2024cs.RO cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Minigolf is an exemplary real-world game for examining embodied intelligence, requiring challenging spatial and kinodynamic understanding to putt the ball. Additionally, reflective reasoning is required if the feasibility of a challenge is not ensured. We introduce RoboGolf, a VLM-based framework that combines dual-camera perception with closed-loop action refinement, augmented by a reflective equilibrium loop. The core of both loops is powered by finetuned VLMs. We analyze the capabilities of the framework in an offline inference setting, relying on an extensive set of recorded trajectories. Exemplary demonstrations of the analyzed problem domain are available at https://jity16.github.io/RoboGolf/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 56 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

42 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 39

  1. 2023Segment Anythingcited by 8,234
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  7. 2023Mastering Diverse Domains through World Modelscited by 1,253
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  11. 2019Event-based Vision: A Surveycited by 1,016
  12. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
27 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  6. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  7. 2019High Speed and High Dynamic Range Video with an Event Cameracited by 345
  8. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  9. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  10. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  11. 2023Large Language Models for Robotics: A Surveycited by 133
  12. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  13. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  14. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 99
  15. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 89
  16. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  17. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  18. 2018ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stackingcited by 55
  19. 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
  20. 2023Large Language Models for Telecom: Forthcoming Impact on the Industrycited by 44
  21. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  22. 2024An Interactive Agent Foundation Modelcited by 25
  23. 2020Forward Prediction for Physical Reasoningcited by 18
  24. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  25. 2024Diffusion-Based Environment-Aware Trajectory Predictioncited by 14
  26. 2017Learning A Physical Long-term Predictorcited by 5
  27. 2016Long-Term Image Boundary Predictioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
  2. 2026Transferring the Intelligence of VLMs to Robotic Control
  3. 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
  4. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective
  5. 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning
  6. 2025Igniting VLMs toward the Embodied Space
  7. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  8. 2025QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models