# DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces arXiv: https://arxiv.org/abs/2509.13444 ยท HTML: https://mlsanity.com/paper/2509.13444/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. - Builds on 34 of its 94 references that are in the map. - First posted on arXiv 16 Sep 2025. ## Placed in - LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/) ## Builds on (most cited first, 30 of 34) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023) - [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023) - [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025) - [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021) - [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024) - [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024) - ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023) - [AppAgentX: Evolving GUI Agents as Proficient Smartphone Users](https://mlsanity.com/paper/2503.02268/) (2025) - [ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions](https://mlsanity.com/paper/2505.14668/) (2025) - [LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem](https://mlsanity.com/paper/2312.03815/) (2023) - [DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing](https://mlsanity.com/paper/2401.10880/) (2024) - [Vibe coding: programming through conversation with artificial intelligence](https://mlsanity.com/paper/2506.23253/) (2025) - [VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning](https://mlsanity.com/paper/2312.11190/) (2023) - [UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback](https://mlsanity.com/paper/2406.07739/) (2024) - [Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model](https://mlsanity.com/paper/2503.04084/) (2025) - [Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents](https://mlsanity.com/paper/2502.13012/) (2025) - [Working with AI: Measuring the Applicability of Generative AI to Occupations](https://mlsanity.com/paper/2507.07935/) (2025) - [MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions](https://mlsanity.com/paper/2502.16796/) (2025) - [Leveraging Multimodal LLM for Inspirational User Interface Search](https://mlsanity.com/paper/2501.17799/) (2025) - [Empowering Agile-Based Generative Software Development through Human-AI Teamwork](https://mlsanity.com/paper/2407.15568/) (2024) - [Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints](https://mlsanity.com/paper/2001.05424/) (2020) - [MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling](https://mlsanity.com/paper/2405.07090/) (2024) - [Misty: UI Prototyping Through Interactive Conceptual Blending](https://mlsanity.com/paper/2409.13900/) (2024) - [Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts](https://mlsanity.com/paper/2404.02475/) (2024) - [Malleable Overview-Detail Interfaces](https://mlsanity.com/paper/2503.07782/) (2025) - [HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions](https://mlsanity.com/paper/2108.03356/) (2021) - [Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces](https://mlsanity.com/paper/2404.13521/) (2024) - [DynEx: Dynamic Code Synthesis with Structured Design Exploration for Accelerated Exploratory Programming](https://mlsanity.com/paper/2410.00400/) (2024) - [Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoning](https://mlsanity.com/paper/2407.06112/) (2024) ## Built on by (most cited first, 4 of 4) - [Plover: Steering GUI Agents through Plan-Centric Interaction](https://mlsanity.com/paper/2607.15193/) (2026) - ["When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces](https://mlsanity.com/paper/2603.02050/) (2026) - [CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks](https://mlsanity.com/paper/2606.09833/) (2026) - [Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution](https://mlsanity.com/paper/2610.20100/) (2026) ## Abstract (arXiv) Large Language Models are reshaping task automation, yet remain limited in complex, multi-step real-world tasks that require aligning with vague user intent and enabling dynamic user override. From a formative study with 12 participants, we found that end-users actively seek to shape task-oriented interfaces rather than relying on one-shot outputs. To address this, we introduce the human-agent co-generation paradigm, materialized in DuetUI. This LLM-empowered system unfolds alongside task progress through a bidirectional context loop-the agent scaffolds the interface by decomposing the task, while the user's direct manipulations implicitly steer the agent's next generation step. In a technical ablation study and a user study with 24 participants, DuetUI improved task efficiency and interface usability, supporting more seamless human-agent collaboration. Our contributions include the proposal of this novel paradigm, the design of a proof-of-concept DuetUI prototype embodying it, and empirical and technical insights from an initial evaluation of how this bidirectional loop may help align agents with human intent and inform future development.