# From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants

arXiv: https://arxiv.org/abs/2604.05166 · HTML: https://mlsanity.com/paper/2604.05166/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 83 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Builds on (most cited first, 29 of 29)

- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Sparks: Inspiration for Science Writing using Language Models](https://mlsanity.com/paper/2110.07640/) (2021)
- [A Design Space for Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2403.14117/) (2024)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)
- [Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge](https://mlsanity.com/paper/2502.00561/) (2025)
- [The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers](https://mlsanity.com/paper/2101.09157/) (2021)
- [AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances](https://mlsanity.com/paper/2409.11360/) (2024)
- [Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies](https://mlsanity.com/paper/2502.08554/) (2025)
- [Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making](https://mlsanity.com/paper/2403.16812/) (2024)
- [The Value, Benefits, and Concerns of Generative AI-Powered Assistance in Writing](https://mlsanity.com/paper/2403.12004/) (2024)
- [Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting](https://mlsanity.com/paper/2303.03199/) (2023)
- [Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions](https://mlsanity.com/paper/2107.07015/) (2021)
- [A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective](https://mlsanity.com/paper/2304.08795/) (2023)
- [LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia](https://mlsanity.com/paper/2207.02308/) (2022)
- [Neural steering vectors reveal dose and exposure-dependent impacts of human-AI relationships](https://mlsanity.com/paper/2512.01991/) (2025)
- [The Role of Inclusion, Control, and Ownership in Workplace AI-Mediated Communication](https://mlsanity.com/paper/2309.11599/) (2023)
- [System Safety and Artificial Intelligence](https://mlsanity.com/paper/2202.09292/) (2022)
- [To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models](https://mlsanity.com/paper/2412.15584/) (2024)
- [From Pen to Prompt: How Creative Writers Integrate AI into their Writing Practice](https://mlsanity.com/paper/2411.03137/) (2024)
- ["It was 80% me, 20% AI": Seeking Authenticity in Co-Writing with Large Language Models](https://mlsanity.com/paper/2411.13032/) (2024)
- [Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process](https://mlsanity.com/paper/2504.12488/) (2025)
- [Effective Interfaces for Student-Driven Revision Sessions for Argumentative Writing](https://mlsanity.com/paper/2107.07018/) (2021)
- [How Does the Disclosure of AI Assistance Affect the Perceptions of Writing?](https://mlsanity.com/paper/2410.04545/) (2024)
- [Users' Mental Models of Generative AI Chatbot Ecosystems](https://mlsanity.com/paper/2501.19211/) (2025)
- [Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms](https://mlsanity.com/paper/2510.10339/) (2025)
- [Deceptive Patterns of Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2404.09375/) (2024)

## Built on by (most cited first, 1 of 1)

- [PeerPen: AI-Assisted Writing for Online Mental Health Peer Support](https://mlsanity.com/paper/2609.14886/) (2026)

## Abstract (arXiv)

AI-based writing assistants are ubiquitous, yet little is known about how users' mental models shape their use. We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works -- and how they affect control behavior -- how users request, accept, or edit AI suggestions as they write -- and writing outcomes. We primed participants (N = 48) with different system descriptions to induce these mental models before asking them to complete a cover letter writing task using a writing assistant that occasionally offered preconfigured ungrammatical suggestions to test whether the mental models affected participants' critical oversight. We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect: while these participants judged the system as more usable, they also produced letters with more grammatical errors, highlighting a complex relationship between system understanding, trust, and control in contexts that require user oversight of error-prone AI outputs.