# Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback

arXiv: https://arxiv.org/abs/2501.03916 · HTML: https://mlsanity.com/paper/2501.03916/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [Large Language Models for Automated Open-domain Scientific Hypotheses Discovery](https://mlsanity.com/paper/2309.02726/) (2023)
- [Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents](https://mlsanity.com/paper/2410.13185/) (2024)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [Hypothesis Generation with Large Language Models](https://mlsanity.com/paper/2404.04326/) (2024)
- [Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System](https://mlsanity.com/paper/2410.09403/) (2024)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas](https://mlsanity.com/paper/2410.14255/) (2024)
- [Large Language Models are Zero Shot Hypothesis Proposers](https://mlsanity.com/paper/2311.05965/) (2023)
- [Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation](https://mlsanity.com/paper/2407.08940/) (2024)
- [CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer](https://mlsanity.com/paper/2405.03376/) (2024)

## Built on by (most cited first, 25 of 25)

- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing](https://mlsanity.com/paper/2503.04629/) (2025)
- [SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning](https://mlsanity.com/paper/2508.06125/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Can We Predict Before Executing Machine Learning Agents?](https://mlsanity.com/paper/2601.05930/) (2026)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [EvolVE: Evolutionary Search for LLM-based Verilog Generation and Optimization](https://mlsanity.com/paper/2601.18067/) (2026)
- [Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction](https://mlsanity.com/paper/2607.12835/) (2026)
- [The Implementation Lottery: Auditing Idea Reliability in Automated Research](https://mlsanity.com/paper/2607.26587/) (2026)
- [ASI-Bench: At the Dawn of Artificial Superintelligence](https://mlsanity.com/paper/2608.17271/) (2026)
- [SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systems](https://mlsanity.com/paper/2508.11310/) (2025)
- [Enhancing Research Idea Generation through Combinatorial Innovation and Multi-Agent Iterative Search Strategies](https://mlsanity.com/paper/2604.20548/) (2026)
- [Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers](https://mlsanity.com/paper/2609.11117/) (2026)
- [SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents](https://mlsanity.com/paper/2606.08234/) (2026)
- [The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testing](https://mlsanity.com/paper/2608.09855/) (2026)
- [OmniScientist: An Omni-Modal Omni-Discipline AI Scientist](https://mlsanity.com/paper/2608.13558/) (2026)
- [Evolving and Executing Research Plans via Double-Loop Multi-Agent Collaboration](https://mlsanity.com/paper/2510.06761/) (2025)
- [Continuous Knowledge Metabolism: Generating Scientific Hypotheses from Evolving Literature](https://mlsanity.com/paper/2604.12243/) (2026)
- [Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systems](https://mlsanity.com/paper/2608.14667/) (2026)
- [Retrieve, Match, Escalate: Accurate and Scalable Product Linking with VLM-Distilled Cross-Encoders and Agentic VLMs](https://mlsanity.com/paper/2608.25037/) (2026)
- [Bioinfoysis Technical Report](https://mlsanity.com/paper/2609.03871/) (2026)
- [Experimental Experience Modeling for Autonomous Research](https://mlsanity.com/paper/2609.39392/) (2026)
- [RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models](https://mlsanity.com/paper/2609.39551/) (2026)
- [JANUS: Enhancing Long-Horizon Autonomous Research with Co-Evolving World Models](https://mlsanity.com/paper/2610.13194/) (2026)

## Abstract (arXiv)

The scientific research paradigm is undergoing a profound transformation owing to the development of Artificial Intelligence (AI). Recent works demonstrate that various AI-assisted research methods can largely improve research efficiency by improving data analysis, accelerating computation, and fostering novel idea generation. To further move towards the ultimate goal (i.e., automatic scientific research), in this paper, we introduce Dolphin, a closed-loop LLM-driven framework to enhance the automation level of scientific research. Dolphin first generates novel ideas based on feedback from previous experiments and relevant papers ranked by the topic and task attributes. Then, the generated ideas can be implemented using a code template refined and debugged with the designed exception-traceback-guided local code structure. Finally, Dolphin automatically analyzes the results of each idea and feeds the results back to the next round of idea generation. Experiments are conducted on the benchmark datasets of different topics and a subset of MLE-bench. Results show that Dolphin can continuously improve the performance of the input topic in a loop. We highlight that Dolphin can automatically propose methods that are comparable to the state-of-the-art in some tasks such as 3D point classification.