Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability
Not on the map. Why?
Abstract (arXiv)
As the boundaries of human computer interaction expand, Generative AI emerges as a key driver in reshaping user interfaces, introducing new possibilities for personalized, multimodal and cross-platform interactions. This integration reflects a growing demand for more adaptive and intuitive user interfaces that can accommodate diverse input types such as text, voice and video, and deliver seamless experiences across devices. This paper explores the integration of generative AI in modern user interfaces, examining historical developments and focusing on multimodal interaction, cross-platform adaptability and dynamic personalization. A central theme is the interface dilemma, which addresses the challenge of designing effective interactions for multimodal large language models, assessing the trade-offs between graphical, voice-based and immersive interfaces. The paper further evaluates lightweight frameworks tailored for mobile platforms, spotlighting the role of mobile hardware in enabling scalable multimodal AI. Technical and ethical challenges, including context retention, privacy concerns and balancing cloud and on-device processing are thoroughly examined. Finally, the paper outlines future directions such as emotionally adaptive interfaces, predictive AI driven user interfaces and real-time collaborative systems, underscoring generative AI's potential to redefine adaptive user-centric interfaces across platforms.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Nov 2024.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz
- 2023Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models
- 2024LLMs Meet Multimodal Generation and Editing: A Survey
- 2025A Survey of Generative Categories and Techniques in Multimodal Generative Models
- 2026When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions
- 2025Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies
- 2026Reality Copilot: Voice-First Human-AI Collaboration in Mixed Reality Using Large Multimodal Models
- 2024Large Language User Interfaces: Voice Interactive User Interfaces powered by LLMs