mlsanityData to 7 Oct 2026

Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors

arXiv 2402.04210 ↗First posted 6 Feb 2024cs.AI cs.ROcited by 23

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Large-scale generative models are shown to be useful for sampling meaningful candidate solutions, yet they often overlook task constraints and user preferences. Their full power is better harnessed when the models are coupled with external verifiers and the final solutions are derived iteratively or progressively according to the verification feedback. In the context of embodied AI, verification often solely involves assessing whether goal conditions specified in the instructions have been met. Nonetheless, for these agents to be seamlessly integrated into daily life, it is crucial to account for a broader range of constraints and preferences beyond bare task success (e.g., a robot should grasp bread with care to avoid significant deformations). However, given the unbounded scope of robot tasks, it is infeasible to construct scripted verifiers akin to those used for explicit-knowledge tasks like the game of Go and theorem proving. This begs the question: when no sound verifier is available, can we use large vision and language models (VLMs), which are approximately omniscient, as scalable Behavior Critics to catch undesirable robot behaviors in videos? To answer this, we first construct a benchmark that contains diverse cases of goal-reaching yet undesirable robot policies. Then, we comprehensively evaluate VLM critics to gain a deeper understanding of their strengths and failure modes. Based on the evaluation, we provide guidelines on how to effectively utilize VLM critiques and showcase a practical way to integrate the feedback into an iterative process of policy refinement. The dataset and codebase are released at: https://guansuns.github.io/pages/vlm-critic.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 55 of its 64 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

55 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents23 Jan

Other references · 54

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Segment Anythingcited by 8,158
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  11. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
38 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2024Self-Rewarding Language Modelscited by 654
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  11. 2023Learning Interactive Real-World Simulatorscited by 440
  12. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  13. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  14. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  15. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  16. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  17. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  18. 2022TALM: Tool Augmented Language Modelscited by 197
  19. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  20. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  21. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  22. 2023Video Language Planningcited by 160
  23. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  24. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  25. 2023Large Language Models for Robotics: A Surveycited by 132
  26. 2020Visual Imitation Made Easycited by 111
  27. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  28. 2023Shepherd: A Critic for Language Model Generationcited by 96
  29. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 88
  30. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  31. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  32. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  33. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  34. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  35. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  36. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  37. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  38. 2023Learning adaptive planning representations with natural language guidancecited by 36

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 8; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  2. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  4. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  5. 2025A Survey on Large Language Models for Automated Planningcited by 17
  6. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  7. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  8. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  9. 2025OpenGVL -- Benchmarking Visual Temporal Progress for Data Curationcited by 9
  10. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  11. 2024Human-AI Safety: A Descendant of Generative AI and Control Systems Safetycited by 7
  12. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
11 more
  1. 2024MotIF: Motion Instruction Fine-tuningcited by 4
  2. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  3. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  4. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  5. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  6. 2024GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidancecited by 2
  7. 2026Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planningcited by 2
  8. 2026DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Modelscited by 2
  9. 2025MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systemscited by 0
  10. 2026On the Strengths and Weaknesses of Data for Open-set Embodied Assistancecited by 0
  11. 2026StageCraft: Execution Aware Mitigation of Distractor and Obstruction Failures in VLA Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents
  2. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
  3. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  4. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignment
  5. 2026Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
  6. 2025RoVer: Robot Reward Model as Test-Time Verifier for Vision-Language-Action Model
  7. 2025TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Models
  8. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents