mlsanityData to 7 Oct 2026

Users Mispredict Their Own Preferences for AI Writing Assistance

arXiv 2601.04461 ↗First posted 8 Jan 2026cs.CL cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Proactive AI writing assistants need to predict when users want drafting help, yet we lack empirical understanding of what drives preferences. Through a factorial vignette study with 50 participants making 750 pairwise comparisons, we find compositional effort dominates decisions (ρ = 0.597) while urgency shows no predictive power (ρ ≈ 0). More critically, users exhibit a striking perception-behavior gap: they rank urgency first in self-reports despite it being the weakest behavioral driver, representing a complete preference inversion. This misalignment has measurable consequences. Systems designed from users' stated preferences achieve only 57.7% accuracy, underperforming even naive baselines, while systems using behavioral patterns reach significantly higher 61.3% (p < 0.05). These findings demonstrate that relying on user introspection for system design actively misleads optimization, with direct implications for proactive natural language generation (NLG) systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 6 of its 29 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

6 of 29 references.

References · 6

  1. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  2. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  3. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  4. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  5. 2021The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writerscited by 61
  6. 2016Smart Reply: Automated Response Suggestion for Emailcited by 54

Built on by

Not cited yet.