Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs
Not on the map. Why?
Abstract (arXiv)
Safety limitations in service robotics across various industries have raised significant concerns about the need for robust mechanisms ensuring that robots adhere to safe practices, thereby preventing actions that might harm humans or cause property damage. Despite advances, including the integration of Knowledge Graphs (KGs) with Large Language Models (LLMs), challenges in ensuring consistent safety in autonomous robot actions persist. In this paper, we propose a novel integration of Large Language Models with Embodied Robotic Control Prompts (ERCPs) and Embodied Knowledge Graphs (EKGs) to enhance the safety framework for service robots. ERCPs are designed as predefined instructions that ensure LLMs generate safe and precise responses. These responses are subsequently validated by EKGs, which provide a comprehensive knowledge base ensuring that the actions of the robot are continuously aligned with safety protocols, thereby promoting safer operational practices in varied contexts. Our experimental setup involved diverse real-world tasks, where robots equipped with our framework demonstrated significantly higher compliance with safety standards compared to traditional methods. This integration fosters secure human-robot interactions and positions our methodology at the forefront of AI-driven safety innovations in service robotics.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 38 of its 77 references that are in the map. First posted on arXiv 28 May 2024.
Builds on
38 of 77 references.
References · 38
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
- 2022Visual Prompt Tuningcited by 1,681
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
- 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
- 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
26 more
- 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 388
- 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
- 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
- 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
- 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
- 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
- 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 171
- 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
- 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 141
- 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
- 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
- 2019PyTorch-BigGraph: A Large-scale Graph Embedding Systemcited by 119
- 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
- 2023Knowledge Graphs: Opportunities and Challengescited by 70
- 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 69
- 2018Improving Natural Language Inference Using External Knowledge in the Science Questions Domaincited by 48
- 2021JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answeringcited by 40
- 2023Deploying and Evaluating LLMs to Program Service Mobile Robotscited by 36
- 2020Object Handovers: a Review for Roboticscited by 28
- 2021LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Promptingcited by 27
- 2024LLM-based Robot Task Planning with Exceptional Handling for General Purpose Service Robotscited by 16
- 2014RoboBrain: Large-Scale Knowledge Engine for Robotscited by 13
- 2021Patterns for Representing Knowledge Graphs to Communicate Situational Knowledge of Service Robotscited by 7
- 2023Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recoverycited by 6
- 2024Consolidating Trees of Robotic Plans Generated Using Large Language Models to Improve Reliabilitycited by 6
- 2023Dobby: A Conversational Service Robot Driven by GPT-4cited by 4
Built on by
Citing papers per year: 2025: 5; 2026: 2.
7 papers.
- 2025A Survey of Context Engineering for Large Language Modelscited by 126
- 2025Safety Aware Task Planning via Large Language Models in Roboticscited by 24
- 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
- 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
- 2026SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Modelscited by 1
- 2025A Signal Contract for Online Language Grounding and Discovery in Decision-Makingcited by 0
- 2026Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directionscited by 0
- 2026Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directions19 Aug 2026
- 2026SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models21 Apr 2026
- 2025A Survey of Context Engineering for Large Language Models17 Jul 2025
- 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities22 Jun 2025
- 2025A Signal Contract for Online Language Grounding and Discovery in Decision-Making9 Jun 2025
- 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agents12 May 2025
- 2025Safety Aware Task Planning via Large Language Models in Robotics19 Mar 2025