# StepWrite: Adaptive Planning for Speech-Driven Text Generation

arXiv: https://arxiv.org/abs/2508.04011 · HTML: https://mlsanity.com/paper/2508.04011/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 112 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 20 of 20)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models](https://mlsanity.com/paper/2305.18323/) (2023)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)
- [Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models](https://mlsanity.com/paper/2402.11723/) (2024)
- [Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2502.16707/) (2025)
- [Wordcraft: a Human-AI Collaborative Editor for Story Writing](https://mlsanity.com/paper/2107.07430/) (2021)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia](https://mlsanity.com/paper/2207.02308/) (2022)
- [Beyond Summarization: Designing AI Support for Real-World Expository Writing Tasks](https://mlsanity.com/paper/2304.02623/) (2023)
- [Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation](https://mlsanity.com/paper/2401.10838/) (2024)
- ["It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Models](https://mlsanity.com/paper/2307.10811/) (2023)
- [Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones](https://mlsanity.com/paper/1608.07323/) (2016)
- [Eliciting and Analysing Users' Envisioned Dialogues with Perfect Voice Assistants](https://mlsanity.com/paper/2102.13508/) (2021)
- [Understanding and Supporting Formal Email Exchange by Answering AI-Generated Questions](https://mlsanity.com/paper/2502.03804/) (2025)
- [Conveying the Predicted Future to Users: A Case Study of Story Plot Prediction](https://mlsanity.com/paper/2302.09122/) (2023)
- [Multitasking with Alexa Multitasking with Alexa: How Using Intelligent Personal Assistants Impacts Language-based Primary Task Performance](https://mlsanity.com/paper/1907.01925/) (2019)

## Built on by (most cited first, 1 of 1)

- ["I followed what felt right, not what I was told": Autonomy, Coaching, and Recognizing Bias Through AI-Mediated Dialogue](https://mlsanity.com/paper/2603.11274/) (2026)

## Abstract (arXiv)

People frequently use speech-to-text systems to compose short texts with voice. However, current voice-based interfaces struggle to support composing more detailed, contextually complex texts, especially in scenarios where users are on the move and cannot visually track progress. Longer-form communication, such as composing structured emails or thoughtful responses, requires persistent context tracking, structured guidance, and adaptability to evolving user intentions--capabilities that conventional dictation tools and voice assistants do not support. We introduce StepWrite, a large language model-driven voice-based interaction system that augments human writing ability by enabling structured, hands-free and eyes-free composition of longer-form texts while on the move. StepWrite decomposes the writing process into manageable subtasks and sequentially guides users with contextually-aware non-visual audio prompts. StepWrite reduces cognitive load by offloading the context-tracking and adaptive planning tasks to the models. Unlike baseline methods like standard dictation features (e.g., Microsoft Word) and conversational voice assistants (e.g., ChatGPT Advanced Voice Mode), StepWrite dynamically adapts its prompts based on the evolving context and user intent, and provides coherent guidance without compromising user autonomy. An empirical evaluation with 25 participants engaging in mobile or stationary hands-occupied activities demonstrated that StepWrite significantly reduces cognitive load, improves usability and user satisfaction compared to baseline methods. Technical evaluations further confirmed StepWrite's capability in dynamic contextual prompt generation, accurate tone alignment, and effective fact checking. This work highlights the potential of structured, context-aware voice interactions in enhancing hands-free and eye-free communication in everyday multitasking scenarios.