# AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topics arXiv: https://arxiv.org/abs/2308.14608 ยท HTML: https://mlsanity.com/paper/2308.14608/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 31 references that are in the map. - First posted on arXiv 28 Aug 2023. ## Builds on (most cited first, 13 of 13) - [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020) - [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021) - [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021) - [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023) - [ReCOVery: A Multimodal Repository for COVID-19 News Credibility Research](https://mlsanity.com/paper/2006.05557/) (2020) - [In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT](https://mlsanity.com/paper/2304.08979/) (2023) - [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022) - [GraphNLI: A Graph-based Natural Language Inference Model for Polarity Prediction in Online Debates](https://mlsanity.com/paper/2202.08175/) (2022) - [Perils and Challenges of Social Media and Election Manipulation Analysis: The 2018 US Midterms](https://mlsanity.com/paper/1902.00043/) (2019) - [A Graph-Based Context-Aware Model to Understand Online Conversations](https://mlsanity.com/paper/2211.09207/) (2022) - [Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function](https://mlsanity.com/paper/2303.13797/) (2023) - [MediaRank: Computational Ranking of Online News Sources](https://mlsanity.com/paper/1903.07581/) (2019) - [Lady and the Tramp Nextdoor: Online Manifestations of Economic Inequalities in the Nextdoor Social Network](https://mlsanity.com/paper/2304.05232/) (2023) ## Built on by (most cited first, 12 of 12) - [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024) - [Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters](https://mlsanity.com/paper/2405.20413/) (2024) - [Identifying and Improving Disability Bias in GPT-Based Resume Screening](https://mlsanity.com/paper/2402.01732/) (2024) - [CodeMirage: Hallucinations in Code Generated by Large Language Models](https://mlsanity.com/paper/2408.08333/) (2024) - [MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models](https://mlsanity.com/paper/2409.19492/) (2024) - [GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy](https://mlsanity.com/paper/2407.18008/) (2024) - [Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models](https://mlsanity.com/paper/2503.16148/) (2025) - [PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models](https://mlsanity.com/paper/2404.08699/) (2024) - [Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System](https://mlsanity.com/paper/2505.01315/) (2025) - [HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes](https://mlsanity.com/paper/2408.05794/) (2024) - [From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models](https://mlsanity.com/paper/2312.17256/) (2023) - [A Holistic Indicator of Polarization to Measure Online Sexism](https://mlsanity.com/paper/2404.02205/) (2024) ## Abstract (arXiv) The introduction of ChatGPT and the subsequent improvement of Large Language Models (LLMs) have prompted more and more individuals to turn to the use of ChatBots, both for information and assistance with decision-making. However, the information the user is after is often not formulated by these ChatBots objectively enough to be provided with a definite, globally accepted answer. Controversial topics, such as "religion", "gender identity", "freedom of speech", and "equality", among others, can be a source of conflict as partisan or biased answers can reinforce preconceived notions or promote disinformation. By exposing ChatGPT to such debatable questions, we aim to understand its level of awareness and if existing models are subject to socio-political and/or economic biases. We also aim to explore how AI-generated answers compare to human ones. For exploring this, we use a dataset of a social media platform created for the purpose of debating human-generated claims on polemic subjects among users, dubbed Kialo. Our results show that while previous versions of ChatGPT have had important issues with controversial topics, more recent versions of ChatGPT (gpt-3.5-turbo) are no longer manifesting significant explicit biases in several knowledge areas. In particular, it is well-moderated regarding economic aspects. However, it still maintains degrees of implicit libertarian leaning toward right-winged ideals which suggest the need for increased moderation from the socio-political point of view. In terms of domain knowledge on controversial topics, with the exception of the "Philosophical" category, ChatGPT is performing well in keeping up with the collective human level of knowledge. Finally, we see that sources of Bing AI have slightly more tendency to the center when compared to human answers. All the analyses we make are generalizable to other types of biases and domains.