# A paradox of AI fluency

arXiv: https://arxiv.org/abs/2604.25905 · HTML: https://mlsanity.com/paper/2604.25905/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration](https://mlsanity.com/paper/2412.15701/) (2024)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models](https://mlsanity.com/paper/2507.06952/) (2025)
- [Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce](https://mlsanity.com/paper/2506.06576/) (2025)
- [How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations](https://mlsanity.com/paper/2510.22780/) (2025)
- [Potemkin Understanding in Large Language Models](https://mlsanity.com/paper/2506.21521/) (2025)
- [Can LLM-Simulated Practice and Feedback Upskill Human Counselors? A Randomized Study with 90+ Novice Counselors](https://mlsanity.com/paper/2505.02428/) (2025)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [AI Safety Should Prioritize the Future of Work](https://mlsanity.com/paper/2504.13959/) (2025)
- [GLAT: The Generative AI Literacy Assessment Test](https://mlsanity.com/paper/2411.00283/) (2024)
- [A clinically validated framework for auditing AI chatbot behavior in mental health interactions](https://mlsanity.com/paper/2602.01347/) (2026)
- [Invisible failures in human-AI interactions](https://mlsanity.com/paper/2603.15423/) (2026)
- [From G-Factor to A-Factor: Establishing a Psychometric Framework for AI Literacy](https://mlsanity.com/paper/2503.16517/) (2025)
- [Calibrated Trust in Dealing with LLM Hallucinations: A Qualitative Study](https://mlsanity.com/paper/2512.09088/) (2025)
- [The Agency Gap in AI-Supported Writing: How Reactive and Proactive Agent Designs Shape Multimodal Reasoning](https://mlsanity.com/paper/2507.04398/) (2025)
- [On Emergent Social World Models -- Evidence for Functional Integration of Theory of Mind and Pragmatic Reasoning in Language Models](https://mlsanity.com/paper/2602.10298/) (2026)

## Abstract (arXiv)

How much does a user's skill with AI shape what AI actually delivers for them? This question is critical for users, AI product builders, and society at large, but it remains underexplored. Using a richly annotated sample of 27K transcripts from WildChat-4.8M, we show that fluent users take on more complex tasks than novices and adopt a fundamentally different interactional mode: they iterate collaboratively with the AI, refining goals and critically assessing outputs, whereas novices take a passive stance. These differences lead to a paradox of AI fluency: fluent users experience more failures than novices -- but their failures tend to be visible (a direct consequence of their engagement), they are more likely to lead to partial recovery, and they occur alongside greater success on complex tasks. Novices, by contrast, more often experience invisible failures: conversations that appear to end successfully but in fact miss the mark. Taken together, these results reframe what success with AI depends on. Individuals should adopt a stance of active engagement rather than passive acceptance. AI product builders should recognize that they are designing not just model behavior but user behavior; encouraging deep engagement, rather than friction-free experiences, will lead to more success overall. Our code and data are available at https://github.com/bigspinai/bigspin-fluency-outcomes