# The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?

arXiv: https://arxiv.org/abs/2410.04699 · HTML: https://mlsanity.com/paper/2410.04699/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Escalation Risks from Language Models in Military and Diplomatic Decision-Making](https://mlsanity.com/paper/2401.03408/) (2024)
- [Global Voices, Local Biases: Socio-Cultural Prejudices across Languages](https://mlsanity.com/paper/2310.17586/) (2023)
- [Evaluating the Interpretability of Generative Models by Interactive Reconstruction](https://mlsanity.com/paper/2102.01264/) (2021)
- [Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing](https://mlsanity.com/paper/2402.05348/) (2024)

## Built on by (most cited first, 14 of 14)

- [The Bias is in the Details: An Assessment of Cognitive Bias in LLMs](https://mlsanity.com/paper/2509.22856/) (2025)
- [Quantifying Cognitive Bias Induction in LLM-Generated Content](https://mlsanity.com/paper/2507.03194/) (2025)
- [Variance reduction in output from generative AI](https://mlsanity.com/paper/2503.01033/) (2025)
- [Political Ideology Shifts in Large Language Models](https://mlsanity.com/paper/2508.16013/) (2025)
- [Unraveling Human-AI Teaming: A Review and Outlook](https://mlsanity.com/paper/2504.05755/) (2025)
- [Show or Tell? Modeling the evolution of request-making in Human-LLM conversations](https://mlsanity.com/paper/2508.01213/) (2025)
- [ROBoto2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment](https://mlsanity.com/paper/2511.03048/) (2025)
- [Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models](https://mlsanity.com/paper/2505.19621/) (2025)
- [Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation](https://mlsanity.com/paper/2511.04706/) (2025)
- [Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities](https://mlsanity.com/paper/2605.04127/) (2026)
- [Approximating the Mathematical Structure of Psychodynamics](https://mlsanity.com/paper/2511.05580/) (2025)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification](https://mlsanity.com/paper/2605.26070/) (2026)
- [Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing](https://mlsanity.com/paper/2606.15914/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses. However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership. The study, conducted in two stages-Topic Discovery and Topic Assignment-integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis. Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics. However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis.