# Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations

arXiv: https://arxiv.org/abs/2409.08937 · HTML: https://mlsanity.com/paper/2409.08937/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2024.

## Built on by (most cited first, 3 of 3)

- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma](https://mlsanity.com/paper/2603.13294/) (2026)

## Abstract (arXiv)

We investigate the impact of hallucinations and Cognitive Forcing Functions in human-AI collaborative content-grounded data generation, focusing on the use of Large Language Models (LLMs) to assist in generating high quality conversational data. Through a study with 34 users who each completed 8 tasks (n=272), we found that hallucinations significantly reduce data quality. While Cognitive Forcing Functions do not always alleviate these effects, their presence influences how users integrate AI responses. Specifically, we observed emerging reliance behaviors, with users often appending AI-generated responses to their correct answers, even when the AI's suggestions conflicted. This points to a potential drawback of Cognitive Forcing Functions, particularly when AI suggestions are inaccurate. Users who overrelied on AI-generated text produced lower quality data, emphasizing the nuanced dynamics of overreliance in human-LLM collaboration compared to traditional human-AI decision-making.