# Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation

arXiv: https://arxiv.org/abs/2407.08940 · HTML: https://mlsanity.com/paper/2407.08940/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 57 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 30 of 42)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [TALM: Tool Augmented Language Models](https://mlsanity.com/paper/2205.12255/) (2022)

## Built on by (most cited first, 30 of 34)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System](https://mlsanity.com/paper/2410.09403/) (2024)
- [EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery](https://mlsanity.com/paper/2603.08127/) (2026)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discovery](https://mlsanity.com/paper/2602.08990/) (2026)
- [Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback](https://mlsanity.com/paper/2501.03916/) (2025)
- [ReviewRL: Towards Automated Scientific Review with RL](https://mlsanity.com/paper/2508.10308/) (2025)
- [MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search](https://mlsanity.com/paper/2505.19209/) (2025)
- [AdsQA: Towards Advertisement Video Understanding](https://mlsanity.com/paper/2509.08621/) (2025)
- [VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities](https://mlsanity.com/paper/2412.18161/) (2024)
- [HypER: Literature-grounded Hypothesis Generation and Distillation with Provenance](https://mlsanity.com/paper/2506.12937/) (2025)
- [Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models](https://mlsanity.com/paper/2505.14599/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [Benchmarking AI scientists for omics data driven biological discovery](https://mlsanity.com/paper/2505.08341/) (2025)
- [Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network](https://mlsanity.com/paper/2511.02238/) (2025)
- [Iterative Hypothesis Generation for Scientific Discovery with Monte Carlo Nash Equilibrium Self-Refining Trees](https://mlsanity.com/paper/2503.19309/) (2025)
- [AInsteinBench: Benchmarking Coding Agents on Scientific Repositories](https://mlsanity.com/paper/2512.21373/) (2025)
- [From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models](https://mlsanity.com/paper/2505.21935/) (2025)
- [BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation](https://mlsanity.com/paper/2508.01285/) (2025)
- [Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs](https://mlsanity.com/paper/2604.08016/) (2026)
- [AI-Driven Automation Can Become the Foundation of Next-Era Science of Science Research](https://mlsanity.com/paper/2505.12039/) (2025)
- [Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning](https://mlsanity.com/paper/2511.21075/) (2025)
- ["Excuse me, may I say something..." CoLabScience, A Proactive AI Assistant for Biomedical Discovery and LLM-Expert Collaborations](https://mlsanity.com/paper/2604.15588/) (2026)
- [Experiment-as-Code Labs: A Declarative Stack for AI-Driven Scientific Discovery](https://mlsanity.com/paper/2605.04375/) (2026)
- [InterFeat: A Pipeline for Finding Interesting Scientific Features](https://mlsanity.com/paper/2505.13534/) (2025)
- [Integrating Dynamical Systems Learning with Foundational Models: A Meta-Evolutionary AI Framework for Clinical Trials](https://mlsanity.com/paper/2506.14782/) (2025)
- [LLM Jaggedness Unlocks Scientific Creativity](https://mlsanity.com/paper/2605.10574/) (2026)
- [DN-Hypo-Pipeline: An AI-Driven Workflow for Generating Hypotheses using Large Language Models and Scientific Explanations](https://mlsanity.com/paper/2606.08532/) (2026)

## Abstract (arXiv)

The rapid growth of biomedical knowledge has outpaced our ability to efficiently extract insights and generate novel hypotheses. Large language models (LLMs) have emerged as a promising tool to revolutionize knowledge interaction and potentially accelerate biomedical discovery. In this paper, we present a comprehensive evaluation of LLMs as biomedical hypothesis generators. We construct a dataset of background-hypothesis pairs from biomedical literature, carefully partitioned into training, seen, and unseen test sets based on publication date to mitigate data contamination. Using this dataset, we assess the hypothesis generation capabilities of top-tier instructed models in zero-shot, few-shot, and fine-tuning settings. To enhance the exploration of uncertainty, a crucial aspect of scientific discovery, we incorporate tool use and multi-agent interactions in our evaluation framework. Furthermore, we propose four novel metrics grounded in extensive literature review to evaluate the quality of generated hypotheses, considering both LLM-based and human assessments. Our experiments yield two key findings: 1) LLMs can generate novel and validated hypotheses, even when tested on literature unseen during training, and 2) Increasing uncertainty through multi-agent interactions and tool use can facilitate diverse candidate generation and improve zero-shot hypothesis generation performance. However, we also observe that the integration of additional knowledge through few-shot learning and tool use may not always lead to performance gains, highlighting the need for careful consideration of the type and scope of external knowledge incorporated. These findings underscore the potential of LLMs as powerful aids in biomedical hypothesis generation and provide valuable insights to guide further research in this area.