# Illusion of Alignment: Detecting Hidden Disagreement in Collaborative Dialogue

arXiv: https://arxiv.org/abs/2608.08210 · HTML: https://mlsanity.com/paper/2608.08210/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 77 references that are in the map.
- First posted on arXiv 8 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://mlsanity.com/paper/2311.10227/) (2023)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Navigating Rifts in Human-LLM Grounding: Study and Benchmark](https://mlsanity.com/paper/2503.13975/) (2025)
- [DeliData: A dataset for deliberation in multi-party problem solving](https://mlsanity.com/paper/2108.05271/) (2021)
- [I Beg to Differ: A study of constructive disagreement in online conversations](https://mlsanity.com/paper/2101.10917/) (2021)
- [The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions](https://mlsanity.com/paper/2406.12480/) (2024)
- [Common Ground Tracking in Multimodal Dialogue](https://mlsanity.com/paper/2403.17284/) (2024)
- [Frictional Agent Alignment Framework: Slow Down and Don't Break Things](https://mlsanity.com/paper/2505.19428/) (2025)
- [Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units](https://mlsanity.com/paper/2403.16609/) (2024)
- [TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues](https://mlsanity.com/paper/2503.09511/) (2025)
- [Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systems](https://mlsanity.com/paper/2501.17348/) (2025)
- [Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations](https://mlsanity.com/paper/2311.01273/) (2023)
- [Any Other Thoughts, Hedgehog? Linking Deliberation Chains in Collaborative Dialogues](https://mlsanity.com/paper/2410.19301/) (2024)
- [Cross-Target Stance Detection: A Survey of Techniques, Datasets, and Challenges](https://mlsanity.com/paper/2409.13594/) (2024)
- [Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs](https://mlsanity.com/paper/2503.12370/) (2025)
- [Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind](https://mlsanity.com/paper/2602.13832/) (2026)
- [Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction](https://mlsanity.com/paper/2603.23531/) (2026)

## Abstract (arXiv)

Collaborative dialogue can end with apparent agreement while participants still differ on goals, assumptions, or execution plans, creating an illusion of alignment (IoA). A real-user study across 18 meetings confirms that IoA arises routinely in human collaboration. Yet IoA poses a paradox: if participants were aware of such disagreements, they would already be explicit; if not, they cannot articulate them when asked, leaving IoA invisible to both participants and observers. In this work, we make IoA detectable by generating diagnostic multiple-choice questions whose divergent answers across participants provide direct behavioral evidence of hidden disagreement. We construct IoA-Suite, a dataset and evaluation protocol for detecting hidden disagreement, spanning five task types and six domains. We find that even the best model attains only 49.5% F1, with the bottleneck traced to private context that the dialogue does not surface. We then train IoA-Prober-8B based on IoA-Suite, reaching 51.8% F1 on IoA-Suite. Across the aforementioned 18 real meetings, it surfaces 2.89 hidden disagreements per meeting that participants confirm they had not voiced, transferring to live human dialogue. Further, in multi-agent collaboration, pairing IoA-Prober-8B with LLM agents improves downstream task performance on BigCodeBench-Hard and HiddenBench.