# Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods

arXiv: https://arxiv.org/abs/2606.26130 · HTML: https://mlsanity.com/paper/2606.26130/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 72 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [Kosmos: An AI Scientist for Autonomous Discovery](https://mlsanity.com/paper/2511.02824/) (2025)
- [Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2503.08979/) (2025)
- [MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://mlsanity.com/paper/2408.14033/) (2024)
- [LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions](https://mlsanity.com/paper/2411.05025/) (2024)
- [Large Language Models are Zero Shot Hypothesis Proposers](https://mlsanity.com/paper/2311.05965/) (2023)
- [The Denario project: Deep knowledge AI agents for scientific discovery](https://mlsanity.com/paper/2510.26887/) (2025)
- [A Vision for Auto Research with LLM Agents](https://mlsanity.com/paper/2504.18765/) (2025)
- [Benchmarks Saturate When The Model Gets Smarter Than The Judge](https://mlsanity.com/paper/2601.19532/) (2026)
- [How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?](https://mlsanity.com/paper/2504.02767/) (2025)
- [Inducing Sustained Creativity and Diversity in Large Language Models](https://mlsanity.com/paper/2603.19519/) (2026)
- [Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNs](https://mlsanity.com/paper/2601.20704/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to guide research methodology, yet their default methodological tendencies under minimal prompting remain unclear. Here, we prompt GPT-5.1, Gemini 3 Pro, and DeepSeek-V3.2 with an LLM-extracted research question from each of 1,000 recent arXiv computer-science papers and compare the resulting methodology suggestions against a paper-derived experimental inventory. Since we provide only the research question, the differences we measure reflect initial suggestions and not how optimal those suggestions are. We extract structured method features from both sources, map them into a shared taxonomy, and quantify divergence across multiple taxonomy dimensions including model provider, dataset task type, and evaluation metric type. The strongest imbalance appears in provider choice, with Jensen-Shannon divergence about 3-5x larger than any other taxonomy dimension. Other/Academic single-occurrence models are underrepresented by 23-24 percentage points, while reused academic/community models are slightly overrepresented (4-6pp). LLMs also suggest a much narrower range of methods overall: the effective number of model entities contracts from 1,232 to 59-96, and inter-LLM rank correlations (0.55-0.68) generally exceed LLM-to-paper correlations (0.33-0.56), so the distortions are largely shared across models. Popularity baselines, BM25 retrieval calibration, and paper-level similarity tests confirm that the outputs are query-specific responses, but filtered through a narrower set of options. Researchers who rely on LLM suggestions without cross-checking therefore risk narrowing their methodological search space toward a more concentrated default.