# MenuCraft: Interactive Menu System Design with Large Language Models

arXiv: https://arxiv.org/abs/2303.04496 · HTML: https://mlsanity.com/paper/2303.04496/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)
- [Wordcraft: a Human-AI Collaborative Editor for Story Writing](https://mlsanity.com/paper/2107.07430/) (2021)
- [LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia](https://mlsanity.com/paper/2207.02308/) (2022)
- [Toward AI Assistants That Let Designers Design](https://mlsanity.com/paper/2107.13074/) (2021)
- [Predicting Human Performance in Vertical Menu Selection Using Deep Learning](https://mlsanity.com/paper/1803.05073/) (2018)
- [Foraging-based Optimization of Menu Systems](https://mlsanity.com/paper/2005.01292/) (2020)

## Built on by (most cited first, 4 of 4)

- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [UI Layout Generation with LLMs Guided by UI Grammar](https://mlsanity.com/paper/2310.15455/) (2023)
- [GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication](https://mlsanity.com/paper/2502.10678/) (2025)
- [APD-Agents: A Large Language Model-Driven Multi-Agents Collaborative Framework for Automated Page Design](https://mlsanity.com/paper/2511.14101/) (2025)

## Abstract (arXiv)

Menu system design for user interfaces is a challenging task involving many design options and various human factors. For example, one crucial factor that designers need to consider is the semantic and systematic relation of menu commands. However, capturing these relations can be challenging due to limited available resources. Large language models can be helpful in this regard, using their pre-training knowledge to design and refine menu systems. In this paper, we propose MenuCraft, an AI-assisted designer for menu design that enables collaboration between the designer and a dialogue system to design menus. MenuCraft offers an interactive language-based menu design tool that simplifies the menu design process and enables easy customization of design options. MenuCraft supports a variety of interactions through dialog that allows performing in-context learning.