mlsanityData to 14 Oct 2026

PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

arXiv 2402.07872 ↗First posted 12 Feb 2024cs.RO cs.CL cs.CVcited by 190

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. This opens the door to richer interaction with the world, for example robotic control. However, VLMs produce only textual outputs, while robotic control and other spatial tasks require outputting continuous coordinates, actions, or trajectories. How can we enable VLMs to handle such settings without fine-tuning on task-specific data? In this paper, we propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as iterative visual question answering. In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e.g., candidate robot actions, localizations, or trajectories). The VLM then selects the best ones for the task. These proposals are iteratively refined, allowing the VLM to eventually zero in on the best available answer. We investigate PIVOT on real-world robotic navigation, real-world manipulation from images, instruction following in simulation, and additional spatial inference tasks such as localization. We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities. Although current performance is far from perfect, our work highlights potentials and limitations of this new regime and shows a promising approach for Internet-Scale VLMs in robotic and spatial reasoning domains. Website: pivot-prompt.github.io and HuggingFace: https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.

Key facts. Cited by 190 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 69 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

59 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023Language to Rewards for Robotic Skill Synthesis14 Jun
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
38 more
  1. 2016Modeling Context in Referring Expressionscited by 1,063
  2. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2023Large Language Models as Optimizerscited by 864
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  6. 2022A Generalist Agentcited by 808
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  12. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  13. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  14. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  15. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  16. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  17. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  18. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  19. 2022Visual Language Maps for Robot Navigationcited by 365
  20. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  21. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  22. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  23. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  24. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  25. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  26. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  27. 2023Large Language Models as General Pattern Machinescited by 197
  28. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  29. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
  30. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  31. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  32. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  33. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  34. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  35. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  36. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  37. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  38. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 56; 2025: 88; 2026: 46.

190 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  4. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  6. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  7. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  8. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  9. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  10. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  11. 2024RT-H: Action Hierarchies Using Languagecited by 210
  12. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
38 more
  1. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  2. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  3. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  5. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  6. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 69
  7. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 65
  8. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  9. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  10. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  11. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 58
  12. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  13. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  14. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  15. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  16. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Modelscited by 46
  17. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  18. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  19. 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environmentscited by 43
  20. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  21. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  22. 2024ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 38
  23. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  24. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  25. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 33
  26. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 32
  27. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 32
  28. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  29. 2025Visual Test-time Scaling for GUI Agent Groundingcited by 30
  30. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 29
  31. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  32. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 26
  33. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 25
  34. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  35. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 23
  36. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  37. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  38. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20

50 most cited of 190 All 190 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
  2. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  3. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  4. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
  5. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
  6. 2026Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
  7. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
  8. 2026PROBE: Manipulation-Grounded Visual Question Answering with VLM Agents