# From Words to Widgets for Controllable LLM Generation

arXiv: https://arxiv.org/abs/2604.10925 · HTML: https://mlsanity.com/paper/2604.10925/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 83 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 30 of 33)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Algorithms to estimate Shapley value feature attributions](https://mlsanity.com/paper/2207.07605/) (2022)
- [Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering](https://mlsanity.com/paper/2309.17249/) (2023)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Aligning to Thousands of Preferences via System Message Generalization](https://mlsanity.com/paper/2405.17977/) (2024)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces](https://mlsanity.com/paper/2309.14459/) (2023)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping](https://mlsanity.com/paper/2304.07810/) (2023)
- [Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks](https://mlsanity.com/paper/2510.27106/) (2025)
- [Supporting Sensemaking of Large Language Model Outputs at Scale](https://mlsanity.com/paper/2401.13726/) (2024)
- [Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild](https://mlsanity.com/paper/2505.16023/) (2025)
- [Context Steering: Controllable Personalization at Inference Time](https://mlsanity.com/paper/2405.01768/) (2024)
- [CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions](https://mlsanity.com/paper/2508.01674/) (2025)
- [Know Your Audience: The benefits and pitfalls of generating plain language summaries beyond the "general" audience](https://mlsanity.com/paper/2403.04979/) (2024)
- [InChorus: Designing Consistent Multimodal Interactions for Data Visualization on Tablet Devices](https://mlsanity.com/paper/2001.06423/) (2020)
- [Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries](https://mlsanity.com/paper/2411.07237/) (2024)
- [Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances](https://mlsanity.com/paper/2307.01142/) (2023)
- [Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models](https://mlsanity.com/paper/2410.23841/) (2024)
- [Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows](https://mlsanity.com/paper/2502.18737/) (2025)
- [Narrix: Remixing Narrative Strategies from Examples for Story Writing](https://mlsanity.com/paper/2604.07643/) (2026)
- [Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgets](https://mlsanity.com/paper/2503.21394/) (2025)
- [PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation](https://mlsanity.com/paper/2507.14913/) (2025)
- [Show or Tell? Modeling the evolution of request-making in Human-LLM conversations](https://mlsanity.com/paper/2508.01213/) (2025)

## Built on by (most cited first, 5 of 5)

- [Priming, Path-dependence, and Plasticity: Understanding the molding of user-LLM interaction and its implications from (many) chat logs in the wild](https://mlsanity.com/paper/2605.05767/) (2026)
- [Narrative Keyframing for Generative Creative Writing](https://mlsanity.com/paper/2608.10337/) (2026)
- [Designing Proactive Thought Partners for Writing](https://mlsanity.com/paper/2609.01588/) (2026)
- [Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responses](https://mlsanity.com/paper/2606.06788/) (2026)
- [DimSteer: Steering LLM Authoring with Automatically Discovered Stylistic Controls](https://mlsanity.com/paper/2610.04174/) (2026)

## Abstract (arXiv)

Natural language remains the predominant way people interact with large language models (LLMs). However, users often struggle to precisely express and control subjective preferences (e.g., tone, style, and emphasis) through prompting. We propose Malleable Prompting, a new interactive prompting technique for controllable LLM generation. It reifies preference expressions in natural language prompts into GUI widgets (e.g., sliders, dropdowns, and toggles) that users can directly configure to steer generation, while visualizing each control's influence on the output to support attribution and comparison across iterations. To enable this interaction, we introduce an LLM decoding algorithm that modulates the token probability distribution during generation based on preference expressions and their widget values. Through a user study, we show that Malleable Prompting helps participants achieve target preferences more precisely and is perceived as more controllable and transparent than natural language prompting alone.