mlsanityData to 14 Oct 2026

Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents

arXiv 2309.09919 ↗First posted 18 Sep 2023cs.RO cs.AI cs.FLcited by 69

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in large language models (LLMs) have enabled a new research domain, LLM agents, for solving robotics and planning tasks by leveraging the world knowledge and general reasoning abilities of LLMs obtained during pretraining. However, while considerable effort has been made to teach the robot the "dos," the "don'ts" received relatively less attention. We argue that, for any practical usage, it is as crucial to teach the robot the "don'ts": conveying explicit instructions about prohibited actions, assessing the robot's comprehension of these restrictions, and, most importantly, ensuring compliance. Moreover, verifiable safe operation is essential for deployments that satisfy worldwide standards such as ISO 61508, which defines standards for safely deploying robots in industrial factory environments worldwide. Aiming at deploying the LLM agents in a collaborative environment, we propose a queryable safety constraint module based on linear temporal logic (LTL) that simultaneously enables natural language (NL) to temporal constraints encoding, safety violation reasoning and explaining, and unsafe action pruning. To demonstrate the effectiveness of our system, we conducted experiments in VirtualHome environment and on a real robot. The experimental results show that our system strictly adheres to the safety constraints and scales well with complex safety constraints, highlighting its potential for practical utility.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 49 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

24 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  9. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  11. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  12. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
11 more
  1. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  2. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  3. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  4. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  5. 2023NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Modelscited by 59
  6. 2023Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specificationcited by 38
  7. 2019Specification Patterns for Robotic Missionscited by 16
  8. 2019Planning With Uncertain Specifications (PUnS)cited by 11
  9. 2022Little Tricky Logic: Misconceptions in the Understanding of LTLcited by 7
  10. 2020Formalizing and Guaranteeing* Human-Robot Interactioncited by 6
  11. 2022Physically-Feasible Repair of Reactive, Linear Temporal Logic-based, High-Level Taskscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 26; 2026: 28.

69 papers.

Sort:

By area:

  1. 2024Language Models with Conformal Factuality Guaranteescited by 131
  2. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  3. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  4. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  5. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  6. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 41
  7. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  8. 2025Safety Guardrails for LLM-Enabled Robotscited by 35
  9. 2024A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddingscited by 24
  10. 2025Safety Aware Task Planning via Large Language Models in Roboticscited by 24
  11. 2025VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verificationcited by 22
  12. 2024SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Modelscited by 21
38 more
  1. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 20
  2. 2025Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Makingcited by 20
  3. 2025Constrained Discrete Diffusioncited by 17
  4. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agentscited by 17
  5. 2024Verifiably Following Complex Robot Instructions with Foundation Modelscited by 16
  6. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 16
  7. 2023Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agentscited by 14
  8. 2025From Prompts to Protection: Large Language Model-Enabled In-Context Learning for Smart Public Safety UAVcited by 10
  9. 2025Distilling On-device Language Models for Robot Planning with Minimal Human Interventioncited by 10
  10. 2024Affordable Generative Agentscited by 9
  11. 2025Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysiscited by 9
  12. 2024Trust-Oriented Adaptive Guardrails for Large Language Modelscited by 8
  13. 2024Safety Control of Service Robots with LLMs and Embodied Knowledge Graphscited by 7
  14. 2025Understanding the Performance and Power of LLM Inferencing on Edge Acceleratorscited by 7
  15. 2026AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agentscited by 7
  16. 2025SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulatorcited by 6
  17. 2026What Breaks Embodied AI Security:LLM Vulnerabilities, CPS Flaws,or Something Else?cited by 6
  18. 2024Procedural Adherence and Interpretability Through Neuro-Symbolic Generative Agentscited by 5
  19. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5
  20. 2025LogicGuard: Improving Embodied LLM agents through Temporal Logic based Criticscited by 5
  21. 2026HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Taskcited by 5
  22. 2025Graphormer-Guided Task Planning: Beyond Static Rules with LLM Safety Perceptioncited by 4
  23. 2026TAPE: Tool-Guided Adaptive Planning and Constrained Execution in Language Model Agentscited by 4
  24. 2026Contextual Safety Reasoning and Grounding for Open-World Robotscited by 4
  25. 2025Hallucination-Aware Generative Pretrained Transformer for Cooperative Aerial Mobility Controlcited by 3
  26. 2025Unifying Inference-Time Planning Language Generationcited by 3
  27. 2025Language Model as Planner and Formalizer under Constraintscited by 3
  28. 2026Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systemscited by 3
  29. 2024HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teamscited by 2
  30. 2025Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Componentscited by 2
  31. 2025HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agentscited by 2
  32. 2025RRTL: Red Teaming Reasoning Large Language Models in Tool Learningcited by 1
  33. 2025Steering Conceptual Bias via Transformer Latent-Subspace Activationcited by 1
  34. 2026Logic-Based Verification of Task Allocation for LLM-Enabled Multi-Agent Manufacturing Systemscited by 1
  35. 2026SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Modelscited by 1
  36. 2026Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deploymentcited by 1
  37. 2026Semantic Constraint Synthesis for Adaptive Trajectory Optimization via Large Language Modelscited by 1
  38. 2026Learning of Robot Safety Policies via Adversarial Synthetic Scenarioscited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SafePlan: Leveraging Formal Logic and Chain-of-Thought Reasoning for Enhanced Safety in LLM-based Robotic Task Planning
  2. 2025VerifyLLM: LLM-Based Pre-Execution Task Plan Verification for Robots
  3. 2026Linear Temporal Logic Translation via Human-Inspired Self-Constrained Reasoning for Robot Task Specification
  4. 2025Collision- and Reachability-Aware Multi-Robot Control with Grounded LLM Planners
  5. 2025Enforcing Temporal Constraints for LLM Agents
  6. 2024Updating Robot Safety Representations Online from Natural Language Feedback
  7. 2026SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems
  8. 2025Enhancing Reliability in LLM-Integrated Robotic Systems: A Unified Approach to Security and Safety