# GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication

arXiv: https://arxiv.org/abs/2502.10678 · HTML: https://mlsanity.com/paper/2502.10678/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 97 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [Understanding Large-Language Model (LLM)-powered Human-Robot Interaction](https://mlsanity.com/paper/2401.03217/) (2024)
- [Improving Grounded Natural Language Understanding through Human-Robot Dialog](https://mlsanity.com/paper/1903.00122/) (2019)
- [Generative Expressive Robot Behaviors using Large Language Models](https://mlsanity.com/paper/2401.14673/) (2024)
- [UI Layout Generation with LLMs Guided by UI Grammar](https://mlsanity.com/paper/2310.15455/) (2023)
- [Patterns for Representing Knowledge Graphs to Communicate Situational Knowledge of Service Robots](https://mlsanity.com/paper/2101.10602/) (2021)
- [A Survey on End-User Robot Programming](https://mlsanity.com/paper/2105.01757/) (2021)
- [Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming](https://mlsanity.com/paper/2407.20712/) (2024)
- [Neural Semantic Parsing with Anonymization for Command Understanding in General-Purpose Service Robots](https://mlsanity.com/paper/1907.01115/) (2019)
- [Sketching Robot Programs On the Fly](https://mlsanity.com/paper/2302.03088/) (2023)
- [MenuCraft: Interactive Menu System Design with Large Language Models](https://mlsanity.com/paper/2303.04496/) (2023)
- [Designing Interfaces for Human-Computer Communication: An On-Going Collection of Considerations](https://mlsanity.com/paper/2309.02257/) (2023)
- [Understanding On-the-Fly End-User Robot Programming](https://mlsanity.com/paper/2406.00841/) (2024)
- [Combining Ontological Knowledge and Large Language Model for User-Friendly Service Robots](https://mlsanity.com/paper/2410.16804/) (2024)
- [Verbal Programming of Robot Behavior](https://mlsanity.com/paper/1911.09782/) (2019)

## Built on by (most cited first, 1 of 1)

- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)

## Abstract (arXiv)

This work investigates the integration of generative visual aids in human-robot task communication. We developed GenComUI, a system powered by large language models that dynamically generates contextual visual aids (such as map annotations, path indicators, and animations) to support verbal task communication and facilitate the generation of customized task programs for the robot. This system was informed by a formative study that examined how humans use external visual tools to assist verbal communication in spatial tasks. To evaluate its effectiveness, we conducted a user experiment (n = 20) comparing GenComUI with a voice-only baseline. The results demonstrate that generative visual aids, through both qualitative and quantitative analysis, enhance verbal task communication by providing continuous visual feedback, thus promoting natural and effective human-robot communication. Additionally, the study offers a set of design implications, emphasizing how dynamically generated visual aids can serve as an effective communication medium in human-robot interaction. These findings underscore the potential of generative visual aids to inform the design of more intuitive and effective human-robot communication, particularly for complex communication scenarios in human-robot interaction and LLM-based end-user development.