# Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model

arXiv: https://arxiv.org/abs/2503.04084 · HTML: https://mlsanity.com/paper/2503.04084/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2025.

## Built on by (most cited first, 26 of 26)

- [Understanding, Protecting, and Augmenting Human Cognition with Generative AI: A Synthesis of the CHI 2025 Tools for Thought Workshop](https://mlsanity.com/paper/2508.21036/) (2025)
- [Just-In-Time Objectives: A General Approach for Specialized AI Interactions](https://mlsanity.com/paper/2510.14591/) (2025)
- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpages](https://mlsanity.com/paper/2505.22831/) (2025)
- [Interaction as Intelligence: Deep Research With Human-AI Partnership](https://mlsanity.com/paper/2507.15759/) (2025)
- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)
- [Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents](https://mlsanity.com/paper/2509.12049/) (2025)
- [PromptCanvas: Composable Prompting Workspaces Using Dynamic Widgets for Exploration and Iteration in Creative Writing](https://mlsanity.com/paper/2506.03741/) (2025)
- [AlignUI: A Method for Designing LLM-Generated UIs Aligned with User Preferences](https://mlsanity.com/paper/2601.17614/) (2026)
- [Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video](https://mlsanity.com/paper/2601.22013/) (2026)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)
- [Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Study](https://mlsanity.com/paper/2604.07558/) (2026)
- [PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents](https://mlsanity.com/paper/2604.08529/) (2026)
- [AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents](https://mlsanity.com/paper/2604.20279/) (2026)
- [Breaking the Curse of Knowledge: Designing Personalized Jargon Support for Real-Time Online Meetings](https://mlsanity.com/paper/2508.10239/) (2025)
- [Textile IR: A Bidirectional Intermediate Representation for Physics-Aware Fashion CAD](https://mlsanity.com/paper/2601.02792/) (2026)
- [Efficient Personalization of Generative User Interfaces](https://mlsanity.com/paper/2604.09876/) (2026)
- [Alignment has a Fantasia Problem](https://mlsanity.com/paper/2604.21827/) (2026)
- [Communication Policy Evolution for Proactive LLM Agents](https://mlsanity.com/paper/2606.14314/) (2026)
- [Beyond Prompt-to-App: Accountable Translation in Teacher-Facing Agentic Authoring](https://mlsanity.com/paper/2609.04679/) (2026)
- [LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions](https://mlsanity.com/paper/2609.14849/) (2026)
- [Personalizing Personal Health Interfaces: Co-Design with Generative AI](https://mlsanity.com/paper/2609.15046/) (2026)
- [Beyond Static Charts: Can Language and Vision Language Models Generate Interactive Data Visualization Interfaces?](https://mlsanity.com/paper/2609.26208/) (2026)
- [A Benchmarking Framework for Context-aware XR Interfaces](https://mlsanity.com/paper/2609.30466/) (2026)
- [Automating the Application of HCI Principles: Skills for On-Demand UI Construction, the Human-AI Space to Think, and the Future of HCI](https://mlsanity.com/paper/2610.02369/) (2026)
- [VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents](https://mlsanity.com/paper/2610.04706/) (2026)

## Abstract (arXiv)

Unlike static and rigid user interfaces, generative and malleable user interfaces offer the potential to respond to diverse users' goals and tasks. However, current approaches primarily rely on generating code, making it difficult for end-users to iteratively tailor the generated interface to their evolving needs. We propose employing task-driven data models-representing the essential information entities, relationships, and data within information tasks-as the foundation for UI generation. We leverage AI to interpret users' prompts and generate the data models that describe users' intended tasks, and by mapping the data models with UI specifications, we can create generative user interfaces. End-users can easily modify and extend the interfaces via natural language and direct manipulation, with these interactions translated into changes in the underlying model. The technical evaluation of our approach and user evaluation of the developed system demonstrate the feasibility and effectiveness of the proposed generative and malleable UIs.