# Rewriting the Script: Adapting Text Instructions for Voice Interaction

arXiv: https://arxiv.org/abs/2306.09992 · HTML: https://mlsanity.com/paper/2306.09992/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 71 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020)
- [Reasoning about Goals, Steps, and Temporal Ordering with WikiHow](https://mlsanity.com/paper/2009.07690/) (2020)
- [Visual Goal-Step Inference using wikiHow](https://mlsanity.com/paper/2104.05845/) (2021)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text](https://mlsanity.com/paper/1909.04745/) (2019)
- [Goal-Oriented Script Construction](https://mlsanity.com/paper/2107.13189/) (2021)
- [Iris: A Conversational Agent for Complex Tasks](https://mlsanity.com/paper/1707.05015/) (2017)
- [Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data](https://mlsanity.com/paper/2203.07264/) (2022)
- [BiSECT: Learning to Split and Rephrase Sentences with Bitexts](https://mlsanity.com/paper/2109.05006/) (2021)
- [Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval](https://mlsanity.com/paper/2111.09276/) (2021)
- [Eliciting and Analysing Users' Envisioned Dialogues with Perfect Voice Assistants](https://mlsanity.com/paper/2102.13508/) (2021)
- [Small but Mighty: New Benchmarks for Split and Rephrase](https://mlsanity.com/paper/2009.08560/) (2020)
- [Unsupervised Multi-Granularity Summarization](https://mlsanity.com/paper/2201.12502/) (2022)
- [Lessons from Oz: Design Guidelines for Automotive Conversational User Interfaces](https://mlsanity.com/paper/1907.11179/) (2019)

## Built on by (most cited first, 5 of 5)

- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [The Impact of Artificial Intelligence on the Evolution of Digital Education: A Comparative Study of OpenAI Text Generation Tools including ChatGPT, Bing Chat, Bard, and Ernie](https://mlsanity.com/paper/2309.02029/) (2023)
- [Grounded Intuition of GPT-Vision's Abilities with Scientific Images](https://mlsanity.com/paper/2311.02069/) (2023)
- [Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap](https://mlsanity.com/paper/2509.16325/) (2025)
- [Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution](https://mlsanity.com/paper/2609.07594/) (2026)

## Abstract (arXiv)

Voice assistants have sharply risen in popularity in recent years, but their use has been limited mostly to simple applications like music, hands-free search, or control of internet-of-things devices. What would it take for voice assistants to guide people through more complex tasks? In our work, we study the limitations of the dominant approach voice assistants take to complex task guidance: reading aloud written instructions. Using recipes as an example, we observe twelve participants cook at home with a state-of-the-art voice assistant. We learn that the current approach leads to nine challenges, including obscuring the bigger picture, overwhelming users with too much information, and failing to communicate affordances. Instructions delivered by a voice assistant are especially difficult because they cannot be skimmed as easily as written instructions. Alexa in particular did not surface crucial details to the user or answer questions well. We draw on our observations to propose eight ways in which voice assistants can ``rewrite the script'' -- summarizing, signposting, splitting, elaborating, volunteering, reordering, redistributing, and visualizing -- to transform written sources into forms that are readily communicated through spoken conversation. We conclude with a vision of how modern advancements in natural language processing can be leveraged for intelligent agents to guide users effectively through complex tasks.