# ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models

arXiv: https://arxiv.org/abs/2306.09649 · HTML: https://mlsanity.com/paper/2306.09649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 78 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands](https://mlsanity.com/paper/1904.09020/) (2019)
- [Schema2QA: High-Quality and Low-Cost Q&A Agents for the Structured Web](https://mlsanity.com/paper/2001.05609/) (2020)
- [Platform for Situated Intelligence](https://mlsanity.com/paper/2103.15975/) (2021)
- [ThingTalk: An Extensible, Executable Representation Language for Task-Oriented Dialogues](https://mlsanity.com/paper/2203.12751/) (2022)
- [Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications](https://mlsanity.com/paper/2007.09809/) (2020)
- [Trade-offs in the Design of Multimodal Interaction for Older Adults](https://mlsanity.com/paper/2103.05309/) (2021)

## Built on by (most cited first, 3 of 3)

- [Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation](https://mlsanity.com/paper/2502.02780/) (2025)
- [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025)
- [A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants](https://mlsanity.com/paper/2510.06223/) (2025)

## Abstract (arXiv)

By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. Traditional multimodal frameworks require laborious developer work to support rich multimodal commands where the user's multimodal command involves possibly exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework that better separates multimodal input from the computational model to enable developers to create efficient and capable multimodal interfaces with ease. ReactGenie translates multimodal user commands into NLPL (Natural Language Programming Language), a programming language we created, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed NLPL and composes primitives in the computational model to implement complex user commands. As a result, ReactGenie allows easy implementation and unprecedented richness in commands for end-users of multimodal apps. Our evaluation showed that 12 developers can learn and build a nontrivial ReactGenie application in under 2.5 hours on average. In addition, compared with a traditional GUI, end-users can complete tasks faster and with less task load using ReactGenie apps.