# Assessing and Explaining the Persuadability of Large Language Models as Legal Decision Tools

arXiv: https://arxiv.org/abs/2604.26233 · HTML: https://mlsanity.com/paper/2604.26233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 29 Apr 2026.

## Builds on (most cited first, 22 of 22)

- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [Parsing Argumentation Structures in Persuasive Essays](https://mlsanity.com/paper/1604.07370/) (2016)
- [Can GPT-3 Perform Statutory Reasoning?](https://mlsanity.com/paper/2302.06100/) (2023)
- [The Persuasive Power of Large Language Models](https://mlsanity.com/paper/2312.15523/) (2023)
- [Exploring the psychology of LLMs' Moral and Legal Reasoning](https://mlsanity.com/paper/2308.01264/) (2023)
- [Moral Alignment for LLM Agents](https://mlsanity.com/paper/2410.01639/) (2024)
- [Mining Legal Arguments in Court Decisions](https://mlsanity.com/paper/2208.06178/) (2022)
- [Conformity in Large Language Models](https://mlsanity.com/paper/2410.12428/) (2024)
- [Persuasion with Large Language Models: A Survey of Empirical Evidence, Study Methodologies, and Ethical Implications](https://mlsanity.com/paper/2411.06837/) (2024)
- [Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Models](https://mlsanity.com/paper/2503.01829/) (2025)
- [LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models](https://mlsanity.com/paper/2504.10430/) (2025)
- [LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models](https://mlsanity.com/paper/2505.00853/) (2025)
- [LLMs Provide Unstable Answers to Legal Questions](https://mlsanity.com/paper/2502.05196/) (2025)
- [Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD](https://mlsanity.com/paper/2508.17450/) (2025)
- [Among Them: A game-based framework for assessing persuasion capabilities of LLMs](https://mlsanity.com/paper/2502.20426/) (2025)
- [On the Adaptive Psychological Persuasion of Large Language Models](https://mlsanity.com/paper/2506.06800/) (2025)
- [Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion](https://mlsanity.com/paper/2509.21054/) (2025)
- [Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries](https://mlsanity.com/paper/2511.06700/) (2025)

## Abstract (arXiv)

As Large Language Models (LLMs) are proposed as legal decision assistants, and even first-instance decision-makers, across a range of judicial and administrative contexts, it becomes essential to explore how they answer legal questions, and in particular the factors that lead them to decide difficult questions. A specific feature of legal decisions is the need to respond to arguments advanced by contending parties. A legal decision-maker must be able to engage with, and respond to, including through being potentially persuaded by, these arguments. Conversely, they should not be unduly persuadable, deciding cases based on the skills of the advocates rather than the merits of the case. In this paper we explore how frontier open- and closed-weights LLMs respond to legal arguments. We propose a metric to measure persuadability in the trilateral setting in which competing advocates seek to persuade a judge of opposite conclusions. We report original experimental results measuring how far the quality of the advocate making arguments affects the likelihood that a given model will agree with a particular legal point of view. We further examine how far models are capable of distinguishing between stronger and weaker arguments and how far model judgments in this domain are affected by positional bias. Through parallel bilateral trials we show how the trilateral setting changes the demands on judge models, and in turn their apparent persuadability. Finally we examine the specific features of arguments that affect persuasion, including the relative contribution of legal content and rhetorical form, the extent to which model persuasion tracks human expert judgments of argument quality, and the extent to which argument quantity, diversity and type affect persuasive outcomes. Our results have implications for the feasibility of adopting LLMs across legal and administrative settings.