# Persuasiveness of Generated Free-Text Rationales in Subjective Decisions: A Case Study on Pairwise Argument Ranking

arXiv: https://arxiv.org/abs/2406.13905 · HTML: https://mlsanity.com/paper/2406.13905/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial](https://mlsanity.com/paper/2403.14380/) (2024)
- [Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?](https://mlsanity.com/paper/2010.04119/) (2020)
- [Few-Shot Self-Rationalization with Natural Language Prompts](https://mlsanity.com/paper/2111.08284/) (2021)
- [A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis](https://mlsanity.com/paper/1911.11408/) (2019)
- [Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong](https://mlsanity.com/paper/2310.12558/) (2023)
- [Automatic Argument Quality Assessment -- New Datasets and Methods](https://mlsanity.com/paper/1909.01007/) (2019)
- [Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales](https://mlsanity.com/paper/2305.07095/) (2023)
- [Are Large Language Models Reliable Argument Quality Annotators?](https://mlsanity.com/paper/2404.09696/) (2024)
- [Can Language Models Recognize Convincing Arguments?](https://mlsanity.com/paper/2404.00750/) (2024)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [Finding Convincing Arguments Using Scalable Bayesian Preference Learning](https://mlsanity.com/paper/1806.02418/) (2018)

## Built on by (most cited first, 7 of 7)

- [Persuasion with Large Language Models: A Survey of Empirical Evidence, Study Methodologies, and Ethical Implications](https://mlsanity.com/paper/2411.06837/) (2024)
- [Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models](https://mlsanity.com/paper/2412.17128/) (2024)
- [CAP: Evaluation of Persuasive and Creative Image Generation](https://mlsanity.com/paper/2412.10426/) (2024)
- [ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentation](https://mlsanity.com/paper/2505.22076/) (2025)
- [SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification](https://mlsanity.com/paper/2511.09539/) (2025)
- [Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach](https://mlsanity.com/paper/2605.28313/) (2026)
- [A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing](https://mlsanity.com/paper/2606.05330/) (2026)

## Abstract (arXiv)

Generating free-text rationales is among the emergent capabilities of Large Language Models (LLMs). These rationales have been found to enhance LLM performance across various NLP tasks. Recently, there has been growing interest in using these rationales to provide insights for various important downstream tasks. In this paper, we analyze generated free-text rationales in tasks with subjective answers, emphasizing the importance of rationalization in such scenarios. We focus on pairwise argument ranking, a highly subjective task with significant potential for real-world applications, such as debate assistance. We evaluate the persuasiveness of rationales generated by nine LLMs to support their subjective choices. Our findings suggest that open-source LLMs, particularly Llama2-70B-chat, are capable of providing highly persuasive rationalizations, surpassing even GPT models. Additionally, our experiments show that rationale persuasiveness can be improved by controlling its parameters through prompting or through self-refinement.