mlsanityData to 14 Oct 2026

RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain

arXiv 2403.14578 ↗First posted 21 Mar 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) increasingly support applications in a wide range of domains, some with potential high societal impact such as biomedicine, yet their reliability in realistic use cases is under-researched. In this work we introduce the Reliability AssesMent for Biomedical LLM Assistants (RAmBLA) framework and evaluate whether four state-of-the-art foundation LLMs can serve as reliable assistants in the biomedical domain. We identify prompt robustness, high recall, and a lack of hallucinations as necessary criteria for this use case. We design shortform tasks and tasks requiring LLM freeform responses mimicking real-world user interactions. We evaluate LLM performance using semantic similarity with a ground truth response, through an evaluator LLM.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

29 of 39 references.

References · 29

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  10. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  11. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  12. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
17 more
  1. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  2. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  4. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  5. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  6. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  7. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 114
  8. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 109
  9. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 96
  10. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 93
  11. 2022Evaluating Human-Language Model Interactioncited by 92
  12. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  13. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 77
  14. 2023Prompting is not a substitute for probability measurements in large language modelscited by 44
  15. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  16. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
  17. 2021Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERTcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystemcited by 11
  2. 2025Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoningcited by 10
  3. 2025Trusting CHATGPT: how minor tweaks in the prompts lead to major differences in sentiment classificationcited by 0
  4. 2026When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variationscited by 0