# The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers

arXiv: https://arxiv.org/abs/2101.09157 · HTML: https://mlsanity.com/paper/2101.09157/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 80 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Gmail Smart Compose: Real-Time Assisted Writing](https://mlsanity.com/paper/1906.00080/) (2019)
- [Smart Reply: Automated Response Suggestion for Email](https://mlsanity.com/paper/1606.04870/) (2016)
- [Visual Interaction with Deep Learning Models through Collaborative Semantic Inference](https://mlsanity.com/paper/1907.10739/) (2019)
- [Diversifying Reply Suggestions using a Matching-Conditional Variational Autoencoder](https://mlsanity.com/paper/1903.10630/) (2019)
- [Conversations with Documents. An Exploration of Document-Centered Assistance](https://mlsanity.com/paper/2002.00747/) (2020)
- [Counterfactual Language Model Adaptation for Suggesting Phrases](https://mlsanity.com/paper/1710.01799/) (2017)

## Built on by (most cited first, 30 of 61)

- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [A Design Space for Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2403.14117/) (2024)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support](https://mlsanity.com/paper/2203.15144/) (2022)
- [AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances](https://mlsanity.com/paper/2409.11360/) (2024)
- [Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models](https://mlsanity.com/paper/2402.11723/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries](https://mlsanity.com/paper/2208.09323/) (2022)
- [Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting](https://mlsanity.com/paper/2303.03199/) (2023)
- [Beyond the Chat: Executable and Verifiable Text-Editing with LLMs](https://mlsanity.com/paper/2309.15337/) (2023)
- [Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers](https://mlsanity.com/paper/2309.12570/) (2023)
- [Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writing](https://mlsanity.com/paper/2208.00636/) (2022)
- [Supporting Sensemaking of Large Language Model Outputs at Scale](https://mlsanity.com/paper/2401.13726/) (2024)
- [The AI Ghostwriter Effect: When Users Do Not Perceive Ownership of AI-Generated Text But Self-Declare as Authors](https://mlsanity.com/paper/2303.03283/) (2023)
- [Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights](https://mlsanity.com/paper/2403.03506/) (2024)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education](https://mlsanity.com/paper/2307.12267/) (2023)
- [ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models](https://mlsanity.com/paper/2310.00117/) (2023)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [Fairness in Language Models Beyond English: Gaps and Challenges](https://mlsanity.com/paper/2302.12578/) (2023)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing](https://mlsanity.com/paper/2107.04007/) (2021)
- ["It was 80% me, 20% AI": Seeking Authenticity in Co-Writing with Large Language Models](https://mlsanity.com/paper/2411.13032/) (2024)
- [Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process](https://mlsanity.com/paper/2504.12488/) (2025)
- [Authors' Values and Attitudes Towards AI-bridged Scalable Personalization of Creative Language Arts](https://mlsanity.com/paper/2403.00439/) (2024)
- [Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools](https://mlsanity.com/paper/2405.17217/) (2024)
- [Storyfier: Exploring Vocabulary Learning Support with Text Generation Models](https://mlsanity.com/paper/2308.03864/) (2023)

## Abstract (arXiv)

We present an in-depth analysis of the impact of multi-word suggestion choices from a neural language model on user behaviour regarding input and text composition in email writing. Our study for the first time compares different numbers of parallel suggestions, and use by native and non-native English writers, to explore a trade-off of "efficiency vs ideation", emerging from recent literature. We built a text editor prototype with a neural language model (GPT-2), refined in a prestudy with 30 people. In an online study (N=156), people composed emails in four conditions (0/1/3/6 parallel suggestions). Our results reveal (1) benefits for ideation, and costs for efficiency, when suggesting multiple phrases; (2) that non-native speakers benefit more from more suggestions; and (3) further insights into behaviour patterns. We discuss implications for research, the design of interactive suggestion systems, and the vision of supporting writers with AI instead of replacing them.