# SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systems

arXiv: https://arxiv.org/abs/2508.11310 · HTML: https://mlsanity.com/paper/2508.11310/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [WritingBench: A Comprehensive Benchmark for Generative Writing](https://mlsanity.com/paper/2503.05244/) (2025)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research](https://mlsanity.com/paper/2505.19955/) (2025)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing](https://mlsanity.com/paper/2503.04629/) (2025)
- [HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models](https://mlsanity.com/paper/2409.16191/) (2024)
- [Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning](https://mlsanity.com/paper/2404.08680/) (2024)
- [SurveyX: Academic Survey Automation via Large Language Models](https://mlsanity.com/paper/2502.14776/) (2025)
- [Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback](https://mlsanity.com/paper/2501.03916/) (2025)
- [Long Text and Multi-Table Summarization: Dataset and Method](https://mlsanity.com/paper/2302.03815/) (2023)
- [A Vision for Auto Research with LLM Agents](https://mlsanity.com/paper/2504.18765/) (2025)
- [Generating a Structured Summary of Numerous Academic Papers: Dataset and Method](https://mlsanity.com/paper/2302.04580/) (2023)
- [System for systematic literature review using multiple AI agents: Concept and an empirical evaluation](https://mlsanity.com/paper/2403.08399/) (2024)
- [InteractiveSurvey: An LLM-based Personalized and Interactive Survey Paper Generation System](https://mlsanity.com/paper/2504.08762/) (2025)
- [Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation](https://mlsanity.com/paper/2411.18583/) (2024)
- [PROMPTHEUS: A Human-Centered Pipeline to Streamline SLRs with LLMs](https://mlsanity.com/paper/2410.15978/) (2024)

## Built on by (most cited first, 3 of 3)

- [SurveyLens: A Discipline-Aware Benchmark for Automatic Survey Generation](https://mlsanity.com/paper/2602.11238/) (2026)
- [OUTLINEFORGE: Hierarchical Reinforcement Learning with Explicit States for Scientific Writing](https://mlsanity.com/paper/2601.09858/) (2026)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)

## Abstract (arXiv)

The growing interest in automatic survey generation (ASG), a task that traditionally required considerable time and effort, has been spurred by recent advances in large language models (LLMs). With advancements in retrieval-augmented generation (RAG) and the rising popularity of multi-agent systems (MASs), synthesizing academic surveys using LLMs has become a viable approach, thereby elevating the need for robust evaluation methods in this domain. However, existing evaluation methods suffer from several limitations, including biased metrics, a lack of human preference, and an over-reliance on LLMs-as-judges. To address these challenges, we propose SGSimEval, a comprehensive benchmark for Survey Generation with Similarity-Enhanced Evaluation that evaluates automatic survey generation systems by integrating assessments of the outline, content, and references, and also combines LLM-based scoring with quantitative metrics to provide a multifaceted evaluation framework. In SGSimEval, we also introduce human preference metrics that emphasize both inherent quality and similarity to humans. Extensive experiments reveal that current ASG systems demonstrate human-comparable superiority in outline generation, while showing significant room for improvement in content and reference generation, and our evaluation metrics maintain strong consistency with human assessments.