# TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models arXiv: https://arxiv.org/abs/2412.15462 ยท HTML: https://mlsanity.com/paper/2412.15462/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 29 of its 55 references that are in the map. - First posted on arXiv 19 Dec 2024. ## Placed in - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 29 of 29) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022) - [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023) - [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021) - [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022) - [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022) - [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023) - [Interactive Language: Talking to Robots in Real Time](https://mlsanity.com/paper/2210.06407/) (2022) - [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023) - [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023) - [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023) - [CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models](https://mlsanity.com/paper/2403.08248/) (2024) - [Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics](https://mlsanity.com/paper/2403.19578/) (2024) - [Language Models as Zero-Shot Trajectory Generators](https://mlsanity.com/paper/2310.11604/) (2023) - [Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review](https://mlsanity.com/paper/2402.10086/) (2024) - [SayTap: Language to Quadrupedal Locomotion](https://mlsanity.com/paper/2306.07580/) (2023) - [RobotGPT: Robot Manipulation Learning from ChatGPT](https://mlsanity.com/paper/2312.01421/) (2023) - [From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control](https://mlsanity.com/paper/2405.04798/) (2024) - [Human-Centric Autonomous Systems With LLMs for User Command Reasoning](https://mlsanity.com/paper/2311.08206/) (2023) - [AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation](https://mlsanity.com/paper/2305.18898/) (2023) - [Lifelong Robot Learning with Human Assisted Language Planners](https://mlsanity.com/paper/2309.14321/) (2023) - [Formal Synthesis of Controllers for Safety-Critical Autonomous Systems: Developments and Challenges](https://mlsanity.com/paper/2402.13075/) (2024) - [The Challenges and Opportunities of Human-Centered AI for Trustworthy Robots and Autonomous Systems](https://mlsanity.com/paper/2105.04408/) (2021) - [Understanding URDF: A Dataset and Analysis](https://mlsanity.com/paper/2308.00514/) (2023) - [Using Formal Methods for Autonomous Systems: Five Recipes for Formal Verification](https://mlsanity.com/paper/2012.00856/) (2020) ## Built on by (most cited first, 4 of 4) - [Keeping it Local, Tiny and Real: Automated Report Generation on Edge Computing Devices for Mechatronic-Based Cognitive Systems](https://mlsanity.com/paper/2511.02507/) (2025) - [Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models](https://mlsanity.com/paper/2504.05477/) (2025) - [Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots](https://mlsanity.com/paper/2509.06768/) (2025) - [The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models](https://mlsanity.com/paper/2607.16318/) (2026) ## Abstract (arXiv) TalkWithMachines aims to enhance human-robot interaction by contributing to interpretable industrial robotic systems, especially for safety-critical applications. The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs), in combination with robotic perception and control. This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs. Moreover, translating the LLM's internal states and reasoning into text that humans can easily understand ensures that operators gain a clearer insight into the robot's current state and intentions, which is essential for effective and safe operation. Our paper outlines four LLM-assisted simulated robotic control workflows, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback, taking the robot's physical capabilities and limitations into account. The proposed concepts are presented in a set of experiments, along with a brief discussion. Project description, videos, and supplementary materials will be available on the project website: https://talk-machines.github.io.