# DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveys

arXiv: https://arxiv.org/abs/2601.15307 · HTML: https://mlsanity.com/paper/2601.15307/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Builds on (most cited first, 20 of 20)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing](https://mlsanity.com/paper/2503.04629/) (2025)
- [MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.00332/) (2024)
- [Artificial Intelligence for the Metaverse: A Survey](https://mlsanity.com/paper/2202.10336/) (2022)
- [You Don't Need Pre-built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures](https://mlsanity.com/paper/2508.06105/) (2025)
- [Think-J: Learning to Think for Generative LLM-as-a-Judge](https://mlsanity.com/paper/2505.14268/) (2025)
- [A Summarization System for Scientific Documents](https://mlsanity.com/paper/1908.11152/) (2019)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [InteractiveSurvey: An LLM-based Personalized and Interactive Survey Paper Generation System](https://mlsanity.com/paper/2504.08762/) (2025)
- [SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation](https://mlsanity.com/paper/2506.12689/) (2025)
- [LiRA: A Multi-Agent Framework for Reliable and Readable Literature Review Generation](https://mlsanity.com/paper/2510.05138/) (2025)
- [SurveyGen-I: Consistent Scientific Survey Generation with Evolving Plans and Memory-Guided Writing](https://mlsanity.com/paper/2508.14317/) (2025)
- [Integrating Randomness in Large Language Models: A Linear Congruential Generator Approach for Generating Clinically Relevant Content](https://mlsanity.com/paper/2407.03582/) (2024)
- [SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection](https://mlsanity.com/paper/2503.03303/) (2025)

## Built on by (most cited first, 4 of 4)

- [Making AI Scientists Auditable from Evidence to Claim](https://mlsanity.com/paper/2606.18874/) (2026)
- [SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators](https://mlsanity.com/paper/2608.07641/) (2026)
- [LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform](https://mlsanity.com/paper/2608.21374/) (2026)
- [How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices](https://mlsanity.com/paper/2610.11877/) (2026)

## Abstract (arXiv)

The rapid development of automated survey generation technology has made it increasingly important to establish a comprehensive benchmark to evaluate the quality of generated surveys. Most existing benchmarks first construct ground-truth datasets by selecting human-written surveys based on limited selection criteria, such as citation counts and structural coherence, and evaluate generated surveys primarily based on conventional quality dimensions, including structural quality and reference relevance. However, these benchmarks have two key issues: (1) the datasets are insufficiently reliable because the selection criteria only identify highly cited or structurally coherent surveys without verifying their academic value; (2) the evaluation metrics mainly reflect the surface-level quality of generated surveys and are insufficient to assess their academic value. Together, these issues prevent existing benchmarks from effectively assessing the academic value of generated surveys. To address the above problems, we propose DeepSurvey-Bench, a comprehensive benchmark for evaluating the academic value of automatically generated surveys. Specifically, our proposed benchmark introduces a set of academic value evaluation criteria covering three dimensions: informational value, scholarly communication value, and research guidance value. We first construct a reliable dataset with academic value annotations based on these criteria, and then evaluate the academic value of generated surveys according to these criteria through a multi-LLM-as-a-judge approach. Extensive experiments demonstrate that DeepSurvey-Bench not only aligns closely with human assessments in evaluating the academic value of surveys, but also reveals underlying academic value beyond the reach of surface-level quality metrics, providing a foundation for fine-grained diagnosis and iterative improvement of generated surveys.