# Robots in the Middle: Evaluating LLMs in Dispute Resolution

arXiv: https://arxiv.org/abs/2410.07053 · HTML: https://mlsanity.com/paper/2410.07053/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://mlsanity.com/paper/2303.00293/) (2023)
- [Explaining Legal Concepts with Augmented Large Language Models (GPT-4)](https://mlsanity.com/paper/2306.09525/) (2023)
- [Diving Deep into Modes of Fact Hallucinations in Dialogue Systems](https://mlsanity.com/paper/2301.04449/) (2023)
- [Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies](https://mlsanity.com/paper/2310.18729/) (2023)
- [From Text to Structure: Using Large Language Models to Support the Development of Legal Expert Systems](https://mlsanity.com/paper/2311.04911/) (2023)
- [Promises and pitfalls of artificial intelligence for legal applications](https://mlsanity.com/paper/2402.01656/) (2024)
- [Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards](https://mlsanity.com/paper/2301.10095/) (2023)
- [LLMediator: GPT-4 Assisted Online Dispute Resolution](https://mlsanity.com/paper/2307.16732/) (2023)

## Built on by (most cited first, 14 of 14)

- [Simulating Dispute Mediation with LLM-Based Agents for Legal Research](https://mlsanity.com/paper/2509.06586/) (2025)
- [Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice](https://mlsanity.com/paper/2412.15260/) (2024)
- [Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs](https://mlsanity.com/paper/2503.12370/) (2025)
- [LLMs as mediators: Can they diagnose conflicts accurately?](https://mlsanity.com/paper/2412.14675/) (2024)
- [Beyond Chat: a Framework for LLMs as Human-Centered Support Systems](https://mlsanity.com/paper/2511.03729/) (2025)
- [LegalWebAgent: Empowering Access to Justice via LLM-Based Web Agents](https://mlsanity.com/paper/2512.04105/) (2025)
- ["I made this (sort of)": Negotiating authorship, confronting fraudulence, and exploring new musical spaces with prompt-based AI music generation](https://mlsanity.com/paper/2507.23365/) (2025)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Generating Place-Based Compromises Between Two Points of View](https://mlsanity.com/paper/2604.24536/) (2026)
- [SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations](https://mlsanity.com/paper/2606.05563/) (2026)
- [Can You Trust What You See? Human and AI Detection of Synthetic Legal Evidence](https://mlsanity.com/paper/2606.07613/) (2026)
- [Automated Mediator for Human Negotiation: Pre-Mediation via a Structured LLM Pipeline](https://mlsanity.com/paper/2606.11379/) (2026)
- [ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation](https://mlsanity.com/paper/2609.11101/) (2026)
- [In With the Old: Enhancing 'Classical' Document Automation with Generative AI](https://mlsanity.com/paper/2610.07480/) (2026)

## Abstract (arXiv)

Mediation is a dispute resolution method featuring a neutral third-party (mediator) who intervenes to help the individuals resolve their dispute. In this paper, we investigate to which extent large language models (LLMs) are able to act as mediators. We investigate whether LLMs are able to analyze dispute conversations, select suitable intervention types, and generate appropriate intervention messages. Using a novel, manually created dataset of 50 dispute scenarios, we conduct a blind evaluation comparing LLMs with human annotators across several key metrics. Overall, the LLMs showed strong performance, even outperforming our human annotators across dimensions. Specifically, in 62% of the cases, the LLMs chose intervention types that were rated as better than or equivalent to those chosen by humans. Moreover, in 84% of the cases, the intervention messages generated by the LLMs were rated as better than or equal to the intervention messages written by humans. LLMs likewise performed favourably on metrics such as impartiality, understanding and contextualization. Our results demonstrate the potential of integrating AI in online dispute resolution (ODR) platforms.