mlsanityData to 7 Oct 2026

Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models

arXiv 2505.07968 ↗First posted 12 May 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have great potential in the field of health care, yet they face great challenges in adapting to rapidly evolving medical knowledge. This can lead to outdated or contradictory treatment suggestions. This study investigated how LLMs respond to evolving clinical guidelines, focusing on concept drift and internal inconsistencies. We developed the DriftMedQA benchmark to simulate guideline evolution and assessed the temporal reliability of various LLMs. Our evaluation of seven state-of-the-art models across 4,290 scenarios demonstrated difficulties in rejecting outdated recommendations and frequently endorsing conflicting guidance. Additionally, we explored two mitigation strategies: Retrieval-Augmented Generation and preference fine-tuning via Direct Preference Optimization. While each method improved model performance, their combination led to the most consistent and reliable results. These findings underscore the need to improve LLM robustness to temporal shifts to ensure more dependable applications in clinical practice. The dataset is available at https://huggingface.co/datasets/RDBH/DriftMed.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 45 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

20 of 45 references.

References · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2024Mixtral of Expertscited by 1,785
8 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  4. 2022Can large language models reason about medical questions?cited by 194
  5. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  6. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  7. 2023Large Language Models to Identify Social Determinants of Health in Electronic Health Recordscited by 55
  8. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Large Language Models Lack Temporal Awareness of Medical Knowledgecited by 2
  2. 2026RAGe: A Retrieval-Augmented Generation Evaluation Frameworkcited by 0
  3. 2026CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
  2. 2025Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
  3. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  4. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?
  5. 2026A primer on evaluation methods for large language models in healthcare
  6. 2026Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation
  7. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
  8. 2025Medical large language models are easily distracted