# DELPHI: Data for Evaluating LLMs' Performance in Handling Controversial Issues

arXiv: https://arxiv.org/abs/2310.18130 · HTML: https://mlsanity.com/paper/2310.18130/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Handling and Presenting Harmful Text in NLP Research](https://mlsanity.com/paper/2204.14256/) (2022)
- [Something's Brewing! Early Prediction of Controversy-causing Posts from Discussion Features](https://mlsanity.com/paper/1904.07372/) (2019)
- [Controversy in Context](https://mlsanity.com/paper/1908.07491/) (2019)
- [Anger Breeds Controversy: Analyzing Controversy and Emotions on Reddit](https://mlsanity.com/paper/2212.00339/) (2022)

## Built on by (most cited first, 11 of 11)

- [LifeTox: Unveiling Implicit Toxicity in Life Advice](https://mlsanity.com/paper/2311.09585/) (2023)
- [LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey](https://mlsanity.com/paper/2402.14558/) (2024)
- [DebateQA: Evaluating Question Answering on Debatable Knowledge](https://mlsanity.com/paper/2408.01419/) (2024)
- [Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning](https://mlsanity.com/paper/2409.15361/) (2024)
- [Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection](https://mlsanity.com/paper/2402.11621/) (2024)
- [Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines](https://mlsanity.com/paper/2512.09483/) (2025)
- [Efficient Safety Retrofitting Against Jailbreaking for LLMs](https://mlsanity.com/paper/2502.13603/) (2025)
- [No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size](https://mlsanity.com/paper/2408.01444/) (2024)
- [Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL](https://mlsanity.com/paper/2503.03654/) (2025)
- [BTPD: A Multilingual Hand-curated Dataset of Bengali Transnational Political Discourse Across Online Communities](https://mlsanity.com/paper/2506.06813/) (2025)
- [Can One-sided Arguments Lead to Response Change in Large Language Models?](https://mlsanity.com/paper/2602.06260/) (2026)

## Abstract (arXiv)

Controversy is a reflection of our zeitgeist, and an important aspect to any discourse. The rise of large language models (LLMs) as conversational systems has increased public reliance on these systems for answers to their various questions. Consequently, it is crucial to systematically examine how these models respond to questions that pertaining to ongoing debates. However, few such datasets exist in providing human-annotated labels reflecting the contemporary discussions. To foster research in this area, we propose a novel construction of a controversial questions dataset, expanding upon the publicly released Quora Question Pairs Dataset. This dataset presents challenges concerning knowledge recency, safety, fairness, and bias. We evaluate different LLMs using a subset of this dataset, illuminating how they handle controversial issues and the stances they adopt. This research ultimately contributes to our understanding of LLMs' interaction with controversial issues, paving the way for improvements in their comprehension and handling of complex societal debates.