# SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators

arXiv: https://arxiv.org/abs/2608.07641 · HTML: https://mlsanity.com/paper/2608.07641/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks](https://mlsanity.com/paper/2508.15804/) (2025)
- [MOPRD: A multidisciplinary open peer review dataset](https://mlsanity.com/paper/2212.04972/) (2022)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining](https://mlsanity.com/paper/2402.15810/) (2024)
- [Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering](https://mlsanity.com/paper/2509.19125/) (2025)
- [Understanding Survey Paper Taxonomy about Large Language Models via Graph Representation Learning](https://mlsanity.com/paper/2402.10409/) (2024)
- [AISysRev -- LLM-based Tool for Title-abstract Screening](https://mlsanity.com/paper/2510.06708/) (2025)
- [DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveys](https://mlsanity.com/paper/2601.15307/) (2026)
- [Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000 post-publication open peer review publishing model](https://mlsanity.com/paper/1911.03379/) (2019)

## Abstract (arXiv)

The rapid advancement of large language models has transformed survey writing from a months-long manual effort into an automated process. As generation scales, reliable evaluation becomes the bottleneck, and LLMs are increasingly used as survey evaluators. However, existing approaches largely rely on off-the-shelf LLM-as-a-judge methods without systematic alignment to human reviewers, and there remains a lack of systematic frameworks for quantifying alignment with human reviewers. To address this gap, we propose SurveyReview, a reviewer-aligned, multi-dimensional benchmark and dataset for survey evaluation. We collect and annotate 675 survey papers with 1,630 review reports. We structure authentic peer-review reports by converting free-form comments into four-dimensional scores (Readability, Criticalness, Comprehensiveness, Structure) paired with supporting rationales. We further release standardized train/test splits and an evaluation protocol to measure alignment between automatic evaluators and human reviewers. To validate the benchmark, we develop SurveyAlign, a strong baseline evaluator by fine-tuning Qwen3-32B with LoRA on our annotated data, augmented with external knowledge for knowledge-intensive dimensions. On the test set, SurveyAlign substantially improves reviewer alignment over prompt-based judging with GPT-5.2, reducing average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 across all four dimensions. Our contributions are twofold: (1) we establish the first multi-dimensional, reviewer-aligned dataset with a reproducible evaluation framework for survey reviewing; (2) we develop a strong baseline evaluator that substantially improves alignment with human reviewers, providing a competitive reference for future research. Our code and data are available at https://surveyreview.github.io