mlsanityData to 7 Oct 2026

What Evidence Do Language Models Find Convincing?

arXiv 2402.11782 ↗First posted 19 Feb 2024cs.CL cs.LGcited by 74

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Retrieval-augmented language models are being increasingly tasked with subjective, contentious, and conflicting queries such as "is aspartame linked to cancer". To resolve these ambiguous queries, one must search through a large range of websites and consider "which, if any, of this evidence do I find convincing?". In this work, we study how LLMs answer this question. In particular, we construct ConflictingQA, a dataset that pairs controversial queries with a series of real-world evidence documents that contain different facts (e.g., quantitative results), argument styles (e.g., appeals to authority), and answers (Yes or No). We use this dataset to perform sensitivity and counterfactual analyses to explore which text features most affect LLM predictions. Overall, we find that current models rely heavily on the relevance of a website to the query, while largely ignoring stylistic features that humans find important such as whether a text contains scientific references or is written with a neutral tone. Taken together, these results highlight the importance of RAG corpus quality (e.g., the need to filter misinformation), and possibly even a shift in how LLMs are trained to better align with human judgements.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 30 of its 50 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

30 of 50 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  12. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
18 more
  1. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  2. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  4. 2023Augmented Language Models: a Surveycited by 398
  5. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  6. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  8. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  9. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  10. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  11. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  12. 2019A Large-scale Dataset for Argument Quality Ranking: Construction and Analysiscited by 66
  13. 2019Automatic Argument Quality Assessment -- New Datasets and Methodscited by 47
  14. 2022Synthetic Disinformation Attacks on Automated Fact Verification Systemscited by 44
  15. 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
  16. 2019Are You Convinced? Choosing the More Convincing Evidence with a Siamese Networkcited by 29
  17. 2022Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systemscited by 11
  18. 2023Answering Ambiguous Questions with a Database of Questions, Answers, and Revisionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 28; 2026: 26.

74 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  3. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  4. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 60
  5. 2024MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debatecited by 54
  6. 2025Do as We Do, Not as You Think: the Conformity of Large Language Modelscited by 48
  7. 2024Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processescited by 38
  8. 2024Adversarial Search Engine Optimization for Large Language Modelscited by 37
  9. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  10. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  11. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  12. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
38 more
  1. 2024Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generationcited by 21
  2. 2024AmbigDocs: Reasoning across Documents on Different Entities under the Same Namecited by 21
  3. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 21
  4. 2024Teaching Models to Balance Resisting and Accepting Persuasioncited by 19
  5. 2024Open Domain Question Answering with Conflicting Contextscited by 18
  6. 2025VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcarecited by 17
  7. 2025Generative Engine Optimization: How to Dominate AI Searchcited by 16
  8. 2024Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steeringcited by 15
  9. 2025Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflictcited by 15
  10. 2024DebateQA: Evaluating Question Answering on Debatable Knowledgecited by 12
  11. 2025ArgRAG: Explainable Retrieval Augmented Generation using Quantitative Bipolar Argumentationcited by 12
  12. 2024Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Stylecited by 11
  13. 2024Open-World Evaluation for Retrieving Diverse Perspectivescited by 11
  14. 2024Generating Media Background Checks for Automated Source Critical Reasoningcited by 10
  15. 2025Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendationscited by 10
  16. 2024A Reality Check on Context Utilisation for Retrieval-Augmented Generationcited by 9
  17. 2026Whose Facts Win? LLM Source Preferences under Knowledge Conflictscited by 8
  18. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?cited by 7
  19. 2025NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledgecited by 7
  20. 2025Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM-based Manipulation Checkscited by 6
  21. 2025Unraveling Misinformation Propagation in LLM Reasoningcited by 6
  22. 2025Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agentscited by 6
  23. 2025How does Misinformation Affect Large Language Model Behaviors and Preferences?cited by 5
  24. 2025Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answeringcited by 5
  25. 2025ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generationcited by 5
  26. 2026Investigating Tool-Memory Conflicts in Tool-Augmented LLMscited by 5
  27. 2025MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collectionscited by 4
  28. 2026Deep-Research Agents Can Be Poisoned via User-Generated Contentcited by 4
  29. 2025CUB: Benchmarking Context Utilisation Techniques for Language Modelscited by 3
  30. 2025Representations of Fact, Fiction and Forecast in Large Language Models: Epistemics and Attitudescited by 3
  31. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  32. 2025CC-GSEO-Bench: A Content-Centric Benchmark for Measuring Source Influence in Generative Search Enginescited by 3
  33. 2025SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Modelscited by 2
  34. 2025Conflicts in Texts: Data, Implications and Challengescited by 2
  35. 2025Building and Measuring Trust between Large Language Modelscited by 2
  36. 2026Rational Synthesizers or Heuristic Followers? Analyzing LLMs in RAG-based Question-Answeringcited by 2
  37. 2026Large Language Models Decide Early and Explain Latercited by 2
  38. 2024Retrieval-Augmented Semantic Parsing: Improving Generalization with Lexical Knowledgecited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
  2. 2025Introducing Answered with Evidence -- a framework for evaluating whether LLM responses to biomedical questions are founded in evidence
  3. 2023Minimizing Factual Inconsistency and Hallucination in Large Language Models
  4. 2026When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering
  5. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals
  6. 2026The strength of clinical evidence is recoverable from language model representations but not from their stated grades
  7. 2026CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine
  8. 2025Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts