mlsanityData to 7 Oct 2026

MedG-KRP: Medical Graph Knowledge Representation Probing

arXiv 2412.10982 ↗First posted 14 Dec 2024cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently emerged as powerful tools, finding many medical applications. LLMs' ability to coalesce vast amounts of information from many sources to generate a response-a process similar to that of a human expert-has led many to see potential in deploying LLMs for clinical use. However, medicine is a setting where accurate reasoning is paramount. Many researchers are questioning the effectiveness of multiple choice question answering (MCQA) benchmarks, frequently used to test LLMs. Researchers and clinicians alike must have complete confidence in LLMs' abilities for them to be deployed in a medical setting. To address this need for understanding, we introduce a knowledge graph (KG)-based method to evaluate the biomedical reasoning abilities of LLMs. Essentially, we map how LLMs link medical concepts in order to better understand how they reason. We test GPT-4, Llama3-70b, and PalmyraMed-70b, a specialized medical model. We enlist a panel of medical students to review a total of 60 LLM-generated graphs and compare these graphs to BIOS, a large biomedical KG. We observe GPT-4 to perform best in our human review but worst in our ground truth comparison; vice-versa with PalmyraMed, the medical model. Our work provides a means of visualizing the medical reasoning pathways of LLMs so they can be implemented in clinical settings safely and effectively.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 43 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

23 of 43 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  6. 2019Publicly Available Clinical BERT Embeddingscited by 995
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  8. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  9. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  11. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  12. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
11 more
  1. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  2. 2024Towards Conversational Diagnostic AIcited by 79
  3. 2023GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphscited by 57
  4. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  5. 2023Integrating Large Language Model for Improved Causal Discoverycited by 54
  6. 2023Causal Order: The Key to Leveraging Imperfect Experts in Causal Inferencecited by 46
  7. 2023Biomedical knowledge graph-optimized prompt generation for large language modelscited by 30
  8. 2022BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Modelscited by 20
  9. 2022Knowledge Graph Generation From Textcited by 15
  10. 2023Applying Large Language Models for Causal Structure Learning in Non Small Cell Lung Cancercited by 8
  11. 2024Context-Aware Reasoning On Parametric Knowledge for Inferring Causal Variablescited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Beyond Correlation: Towards Causal Large Language Model Agents in Biomedicinecited by 0
  2. 2025SwasthLLM: a Unified Cross-Lingual, Multi-Task, and Meta-Learning Zero-Shot Framework for Medical Diagnosis Using Contrastive Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
  2. 2026ReGraM: Region-First Knowledge Graph Reasoning for Medical Question Answering
  3. 2025MKG-Rank: Enhancing Large Language Models with Knowledge Graph for Multilingual Medical Question Answering
  4. 2025Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge
  5. 2024RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine
  6. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  7. 2023HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responses
  8. 2025Structured Outputs Enable General-Purpose LLMs to be Medical Experts