# Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

arXiv: https://arxiv.org/abs/2412.17596 · HTML: https://mlsanity.com/paper/2412.17596/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 30 of its 83 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas](https://mlsanity.com/paper/2410.14255/) (2024)
- [IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback](https://mlsanity.com/paper/2410.04025/) (2024)
- [Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation](https://mlsanity.com/paper/2409.14634/) (2024)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [Assessing and Understanding Creativity in Large Language Models](https://mlsanity.com/paper/2401.12491/) (2024)
- [SciPIP: An LLM-based Scientific Paper Idea Proposer](https://mlsanity.com/paper/2410.23166/) (2024)
- [Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA](https://mlsanity.com/paper/2408.09235/) (2024)
- [Forecasting high-impact research topics via machine learning on evolving knowledge graphs](https://mlsanity.com/paper/2402.08640/) (2024)
- [Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders](https://mlsanity.com/paper/2405.17044/) (2024)
- [Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation](https://mlsanity.com/paper/2305.10679/) (2023)

## Built on by (most cited first, 28 of 28)

- [AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents](https://mlsanity.com/paper/2602.06855/) (2026)
- [CellForge: Agentic Design of Virtual Cell Models](https://mlsanity.com/paper/2508.02276/) (2025)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Measuring the Gap Between Human and LLM Research Ideas](https://mlsanity.com/paper/2607.01233/) (2026)
- [Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting](https://mlsanity.com/paper/2509.06770/) (2025)
- [PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review](https://mlsanity.com/paper/2601.19916/) (2026)
- [Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation](https://mlsanity.com/paper/2604.28031/) (2026)
- [Auto-Stega: An Agent-Driven System for Lifelong Strategy Evolution in LLM-Based Text Steganography](https://mlsanity.com/paper/2510.06565/) (2025)
- [Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steering](https://mlsanity.com/paper/2601.22010/) (2026)
- [Evolving Idea Graphs with Learnable Edits-and-Commits for Multi-Agent Scientific Ideation](https://mlsanity.com/paper/2605.04922/) (2026)
- [Are LLMs becoming similarly creative? Evidence from three years of models](https://mlsanity.com/paper/2608.19437/) (2026)
- [What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation](https://mlsanity.com/paper/2608.22948/) (2026)
- [SciAtlas: A Computable Atlas of Science for Knowledge-Grounded AI Research](https://mlsanity.com/paper/2605.22878/) (2026)
- [AgentPanel: Toward a New Paradigm for Human--AI Collaboration in Exploring Scientific Questions](https://mlsanity.com/paper/2608.03283/) (2026)
- [Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment](https://mlsanity.com/paper/2608.29696/) (2026)
- [Creative Generation via Multi-Agent Debate: Does Debate Suppress Diversity?](https://mlsanity.com/paper/2609.00683/) (2026)
- [Spacer: Towards Engineered Scientific Inspiration](https://mlsanity.com/paper/2508.17661/) (2025)
- [Universe of Thoughts: Enabling Creative Reasoning with Large Language Models](https://mlsanity.com/paper/2511.20471/) (2025)
- [Continuous Knowledge Metabolism: Generating Scientific Hypotheses from Evolving Literature](https://mlsanity.com/paper/2604.12243/) (2026)
- [Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers](https://mlsanity.com/paper/2605.13450/) (2026)
- [AGC-Bench: Measuring Artificial General Creativity](https://mlsanity.com/paper/2607.01152/) (2026)
- [Large language models for partial differential equation workflows](https://mlsanity.com/paper/2608.03600/) (2026)
- [Hallucination as a Feature, not a Defect: Evaluating a multi-agent architecture to transform speculative language-model outputs into testable scientific hypotheses](https://mlsanity.com/paper/2608.19206/) (2026)
- [Answer Probing-Guided Search for Diverse Solution Exploration of LLMs](https://mlsanity.com/paper/2608.30345/) (2026)
- [Is Imagination Derived from Hallucination? A Cross-Taxonomy Evaluation of Imagination and Hallucination in Large Language Models](https://mlsanity.com/paper/2609.22152/) (2026)
- [IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation](https://mlsanity.com/paper/2610.04074/) (2026)
- [Priced Guidance: Can Language Models Generate Future Research Ideas?](https://mlsanity.com/paper/2610.04976/) (2026)

## Abstract (arXiv)

While Large Language Models (LLMs) demonstrate remarkable capabilities in scientific tasks such as literature analysis and experimental design (e.g., accurately extracting key findings from papers or generating coherent experimental procedures), existing evaluation benchmarks primarily assess performance using rich contextual inputs. We introduce LiveIdeaBench, a comprehensive benchmark evaluating LLMs' scientific idea generation by assessing divergent thinking capabilities using single-keyword prompts. Drawing from Guilford's creativity theory, our benchmark employs a dynamic panel of state-of-the-art LLMs to assess generated ideas across five key dimensions: originality, feasibility, fluency, flexibility, and clarity. Through extensive experimentation with over 40 leading models across 1,180 keywords spanning 22 scientific domains, we reveal that the scientific idea generation capabilities measured by our benchmark, are poorly predicted by standard metrics of general intelligence. Our results demonstrate that models like QwQ-32B-preview achieve creative performance comparable to top-tier models such as claude-3.7-sonnet:thinking, despite significant gaps in their general intelligence scores. These findings highlight the need for specialized evaluation benchmarks for scientific idea generation and suggest that enhancing these idea generation capabilities in LLMs may require different training strategies than those used for improving general problem-solving abilities, potentially enabling a wider range of AI tools tailored for different stages of the scientific process.