# When the AI Leaves the Tailorshop: Measuring What an LLM Advisor Leaves Behind in Complex Problem Solving

arXiv: https://arxiv.org/abs/2610.00163 · HTML: https://mlsanity.com/paper/2610.00163/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 99 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024)
- [The Metacognitive Demands and Opportunities of Generative AI](https://mlsanity.com/paper/2312.10893/) (2023)
- [Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2301.11333/) (2023)
- [Toward a principled Bayesian workflow in cognitive science](https://mlsanity.com/paper/1904.12765/) (2019)
- [Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skills](https://mlsanity.com/paper/2410.04253/) (2024)
- [Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration](https://mlsanity.com/paper/2402.18498/) (2024)
- [Gender, Age, and Technology Education Influence the Adoption and Appropriation of LLMs](https://mlsanity.com/paper/2310.06556/) (2023)
- [AI Assistance Reduces Persistence and Hurts Independent Performance](https://mlsanity.com/paper/2604.04721/) (2026)
- [P-value: A Bless or A Curse for Evidence-Based Studies?](https://mlsanity.com/paper/1809.08503/) (2018)
- [Metacognition and Confidence Dynamics in Advice Taking from Generative AI](https://mlsanity.com/paper/2510.26508/) (2025)

## Abstract (arXiv)

Complex problem solving depends on acting effectively and understanding how a system works. AI advice may support these outcomes unequally. Two preregistered experiments compared participants managing a simulated clothing factory with and without an LLM advisor. Across studies, AI-supported participants reported greater confidence and understanding with less effort. In the first study (N=200), assistance increased company value but produced no detectable prediction-accuracy difference. After withdrawal, previously supported participants outperformed controls when decisions were scored against repeating previous choices, but not default settings. Within the AI-supported group, more frequent recommendation alterations predicted better unaided performance. In the second study (N=198), AI-supported participants went bankrupt less often and showed a small knowledge advantage in the registered analysis, largely associated with remaining solvent. More frequent recommendation alterations predicted higher knowledge within the AI-supported group. Applied HAI evaluation should assess users' understanding and independent capability alongside the performance achieved with AI support.