mlsanityData to 7 Oct 2026

Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection

arXiv 2412.04455 ↗First posted 5 Dec 2024cs.RO cs.AI cs.CVcited by 59

Not on the map. Why?

Abstract (arXiv)

Automatic detection and prevention of open-set failures are crucial in closed-loop robotic systems. Recent studies often struggle to simultaneously identify unexpected failures reactively after they occur and prevent foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a novel paradigm leveraging the vision-language model (VLM) for both open-set reactive and proactive failure detection. The core of our method is to formulate both tasks as a unified set of spatio-temporal constraint satisfaction problems and use VLM-generated code to evaluate them for real-time monitoring. To enhance the accuracy and efficiency of monitoring, we further introduce constraint elements that abstract constraint-related entities or their parts into compact geometric elements. This approach offers greater generality, simplifies tracking, and facilitates constraint-aware visual programming by leveraging these elements as visual prompts. Experiments show that CaM achieves a 28.7% higher success rate and reduces execution time by 31.8% under severe disturbances compared to baselines across three simulators and a real-world setting. Moreover, CaM can be integrated with open-loop control policies to form closed-loop systems, enabling long-horizon tasks in cluttered scenes with dynamic environments.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 67 of its 88 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

67 of 88 references.

References · 67

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  5. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  6. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  12. 2023Segment Everything Everywhere All at Oncecited by 546
  13. 2023CoTracker: It is Better to Track Togethercited by 516
  14. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  15. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  16. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  17. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  18. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  19. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  20. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  21. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  22. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  23. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  24. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  25. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  26. 2023PACO: Parts and Attributes of Common Objectscited by 146
  27. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  28. 2023Vision-Language Models as Success Detectorscited by 138
  29. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  30. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  31. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  32. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  33. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  34. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  35. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  36. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 59
  37. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  38. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 31.

59 papers.

Sort:

By area:

  1. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  2. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Roboticscited by 125
  3. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  4. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  5. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  6. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  7. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 26
  8. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  9. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  10. 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidancecited by 17
  11. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  12. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
38 more
  1. 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 11
  2. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 11
  3. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9
  4. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  5. 2026Self-Refining Vision Language Model for Robotic Failure Detection and Reasoningcited by 6
  6. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Roboticscited by 6
  7. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 6
  8. 2026PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Executioncited by 5
  9. 2026VLCP: Vision Language Control Policy Closed-Loop Code Replanning for Robot Manipulationcited by 5
  10. 2025A Unified Framework for Real-Time Failure Handling in Robotics Using Vision-Language Models, Reactive Planner and Behavior Treescited by 4
  11. 2025MathSticks: A Benchmark for Visual Symbolic Compositional Reasoning with Matchstick Puzzlescited by 4
  12. 2026Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recoverycited by 4
  13. 2025HybridGen: VLM-Guided Hybrid Planning for Scalable Data Generation of Imitation Learningcited by 3
  14. 2025LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledgecited by 3
  15. 2025A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Roboticscited by 2
  16. 2025EmbodiedCoder: Parameterized Embodied Mobile Manipulation via Modern Coding Modelcited by 2
  17. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  18. 2025Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Interventioncited by 2
  19. 2026ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agentscited by 2
  20. 2026PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulationcited by 2
  21. 2026LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agentscited by 2
  22. 2026CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environmentcited by 1
  23. 2026VISOR: A Vision-Language Model-based Test Oracle for Testing Robotscited by 1
  24. 2026CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RLcited by 1
  25. 2026RoboBRIDGE: A Modular Framework for Bridging Policies to Robust Real-World Robotic Agentscited by 1
  26. 2025Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulationcited by 0
  27. 2025FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Modelscited by 0
  28. 2025BINDER: Instantly Adaptive Mobile Manipulation with Open-Vocabulary Commandscited by 0
  29. 2025GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Taskscited by 0
  30. 2026HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigationcited by 0
  31. 2026NEXUS: Continual Learning of Symbolic Constraints for Safe and Robust Embodied Planningcited by 0
  32. 2026GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logiccited by 0
  33. 2026LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratorycited by 0
  34. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0
  35. 2026AGM: Achievement-Grounded Memory for Closed-Loop Agents with Frozen VLA Policiescited by 0
  36. 2026The Rise of Verbal Reinforcement Learningcited by 0
  37. 2026FailBench: How Reliable are VLMs at Judging Robot Task Success?cited by 0
  38. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Addressing Failures in Robotics using Vision-Based Language Models (VLMs) and Behavior Trees (BT)
  2. 2026FailureSpot: Label-Efficient Timestamp-Level Failure Detection for Vision-Language-Action Models
  3. 2026Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring
  4. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
  5. 2026Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents
  6. 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
  7. 2026VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models
  8. 2026KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis