# Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs

arXiv: https://arxiv.org/abs/2605.29928 · HTML: https://mlsanity.com/paper/2605.29928/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 76 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [AI model GPT-3 (dis)informs us better than humans](https://mlsanity.com/paper/2301.11924/) (2023)
- [Human-Centered Design Recommendations for LLM-as-a-Judge](https://mlsanity.com/paper/2407.03479/) (2024)
- [Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Media](https://mlsanity.com/paper/2412.18148/) (2024)
- [How susceptible are LLMs to Logical Fallacies?](https://mlsanity.com/paper/2308.09853/) (2023)
- [Labeling Messages as AI-Generated Does Not Reduce Their Persuasive Effects](https://mlsanity.com/paper/2504.09865/) (2025)
- [Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments](https://mlsanity.com/paper/2212.07425/) (2022)
- [Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinations](https://mlsanity.com/paper/2404.03745/) (2024)
- [How Does the Disclosure of AI Assistance Affect the Perceptions of Writing?](https://mlsanity.com/paper/2410.04545/) (2024)
- [Large Language Models Are Better Logical Fallacy Reasoners with Counterargument, Explanation, and Goal-Aware Prompt Formulation](https://mlsanity.com/paper/2503.23363/) (2025)
- [Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings](https://mlsanity.com/paper/2109.07833/) (2021)
- [CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowds](https://mlsanity.com/paper/2410.03457/) (2024)
- [LLM-C3MOD: A Human-LLM Collaborative System for Cross-Cultural Hate Speech Moderation](https://mlsanity.com/paper/2503.07237/) (2025)
- [Penalizing Transparency? How AI Disclosure and Author Demographics Shape Human and AI Judgments About Writing](https://mlsanity.com/paper/2507.01418/) (2025)
- [Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning](https://mlsanity.com/paper/2509.18163/) (2025)
- [Catch Me if You Search: When Contextual Web Search Results Affect the Detection of Hallucinations](https://mlsanity.com/paper/2504.01153/) (2025)
- [Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection](https://mlsanity.com/paper/2510.09695/) (2025)
- [Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge](https://mlsanity.com/paper/2604.05593/) (2026)
- [Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated](https://mlsanity.com/paper/2410.03723/) (2024)
- [Effect of AI Performance, Risk Perception, and Trust on Human Dependence in Deepfake Detection AI system](https://mlsanity.com/paper/2508.01906/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Argument and the Letterhead: Source-Position Coherence in AI Evaluation](https://mlsanity.com/paper/2609.35286/) (2026)

## Abstract (arXiv)

As AI-generated and AI-assisted content floods online spaces, source labels attached to such content can distort human reasoning judgments, with downstream consequences for moderation, evaluation, and decision-making. Whether LLMs share this vulnerability, or offer more source-agnostic evaluation, remains an open question with strong implications for human-AI collaboration. We examine this issue using logical fallacies as a controlled setting to isolate source-label effects on reasoning quality, independent of domain knowledge. We conduct an online study (N=505) where participants are assigned to a source condition (human, AI, human with AI assistance, AI with human assistance, or no disclosure) and evaluate comments containing logical fallacies, comparing their judgments with those of LLMs (GPT-5.2, Gemini 2.5 Flash, Claude Sonnet 4.5), which were evaluated across the same source conditions. Human evaluators were significantly more susceptible to fallacies labeled as 'written by human' or 'written by human with AI assistance' and assigned higher trust ratings in these conditions. LLM evaluations remained comparatively stable across source labels, though performance varied across models. Confidence levels were similarly high across conditions for both humans and LLMs, regardless of the presence of fallacies. Our findings indicate that source-label bias is primarily a human vulnerability for logical fallacy evaluation, with potential implications in human-LLM collaboration in increasingly AI-mediated environments.