# Navigating Rifts in Human-LLM Grounding: Study and Benchmark

arXiv: https://arxiv.org/abs/2503.13975 · HTML: https://mlsanity.com/paper/2503.13975/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 81 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Instruction Tuning With Loss Over Instructions](https://mlsanity.com/paper/2405.14394/) (2024)
- [Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations](https://mlsanity.com/paper/2311.05584/) (2023)
- [Controllable Mixed-Initiative Dialogue Generation through Prompting](https://mlsanity.com/paper/2305.04147/) (2023)
- [Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing](https://mlsanity.com/paper/1807.06517/) (2018)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [Instruction Fine-Tuning: Does Prompt Loss Matter?](https://mlsanity.com/paper/2401.13586/) (2024)
- [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023)
- [ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog](https://mlsanity.com/paper/2409.06097/) (2024)
- [Grounding Conversations with Improvised Dialogues](https://mlsanity.com/paper/2004.09544/) (2020)
- [Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality](https://mlsanity.com/paper/2211.09267/) (2022)
- [Modeling Cross-Cultural Pragmatic Inference with Codenames Duet](https://mlsanity.com/paper/2306.02475/) (2023)
- [Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction](https://mlsanity.com/paper/2011.13137/) (2020)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [Conversation as Action Under Uncertainty](https://mlsanity.com/paper/1301.3883/) (2013)

## Built on by (most cited first, 29 of 29)

- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [Creating General User Models from Computer Use](https://mlsanity.com/paper/2505.10831/) (2025)
- [User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal](https://mlsanity.com/paper/2507.23158/) (2025)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [LVLMs and Humans Ground Differently in Referential Communication](https://mlsanity.com/paper/2601.19792/) (2026)
- [Completion neq Collaboration: Scaling Collaborative Effort with Agents](https://mlsanity.com/paper/2510.25744/) (2025)
- ["What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets](https://mlsanity.com/paper/2506.21532/) (2025)
- [Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale](https://mlsanity.com/paper/2504.09283/) (2025)
- [Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations](https://mlsanity.com/paper/2605.14175/) (2026)
- [Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations](https://mlsanity.com/paper/2510.12699/) (2025)
- [Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone](https://mlsanity.com/paper/2605.04454/) (2026)
- [Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments](https://mlsanity.com/paper/2606.15766/) (2026)
- [Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue](https://mlsanity.com/paper/2606.31719/) (2026)
- [Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue](https://mlsanity.com/paper/2609.12373/) (2026)
- [From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost](https://mlsanity.com/paper/2609.21117/) (2026)
- [The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences](https://mlsanity.com/paper/2509.11295/) (2025)
- [Alignment has a Fantasia Problem](https://mlsanity.com/paper/2604.21827/) (2026)
- [Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations](https://mlsanity.com/paper/2604.22109/) (2026)
- [Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration](https://mlsanity.com/paper/2606.06388/) (2026)
- [Illusion of Alignment: Detecting Hidden Disagreement in Collaborative Dialogue](https://mlsanity.com/paper/2608.08210/) (2026)
- [When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs](https://mlsanity.com/paper/2608.23978/) (2026)
- [How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversations](https://mlsanity.com/paper/2608.29591/) (2026)
- [You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding](https://mlsanity.com/paper/2608.29834/) (2026)
- [Pretrained Persona Mixture Models and Tandem Models for Human Simulation](https://mlsanity.com/paper/2609.22607/) (2026)
- [Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPT](https://mlsanity.com/paper/2609.35953/) (2026)
- [Referential Uncertainty in Human--AI Collaboration](https://mlsanity.com/paper/2609.39518/) (2026)
- [Dataset Signatures in Human-LLM Interactions and User Modeling](https://mlsanity.com/paper/2610.05534/) (2026)
- [Measuring and Reducing Unusable Replies of LLM Agents](https://mlsanity.com/paper/2610.19659/) (2026)

## Abstract (arXiv)

Language models excel at following instructions but often struggle with the collaborative aspects of conversation that humans naturally employ. This limitation in grounding -- the process by which conversation participants establish mutual understanding -- can lead to outcomes ranging from frustrated users to serious consequences in high-stakes scenarios. To systematically study grounding challenges in human-LLM interactions, we analyze logs from three human-assistant datasets: WildChat, MultiWOZ, and Bing Chat. We develop a taxonomy of grounding acts and build models to annotate and forecast grounding behavior. Our findings reveal significant differences in human-human and human-LLM grounding: LLMs were three times less likely to initiate clarification and sixteen times less likely to provide follow-up requests than humans. Additionally, we find that early grounding failures predict later interaction breakdowns. Building on these insights, we introduce Rifts, a benchmark derived from publicly available LLM interaction data containing situations where LLMs fail to initiate grounding. We note that current frontier models perform poorly on Rifts, highlighting the need to reconsider how we train and prompt LLMs for human interaction. To this end, we develop a preliminary intervention aimed at mitigating grounding failures.