# Automatic Evaluation Metrics for Artificially Generated Scientific Research

arXiv: https://arxiv.org/abs/2503.05712 · HTML: https://mlsanity.com/paper/2503.05712/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Large Language Models for Automated Open-domain Scientific Hypotheses Discovery](https://mlsanity.com/paper/2309.02726/) (2023)
- [MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://mlsanity.com/paper/2408.14033/) (2024)
- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [CiteBench: A benchmark for Scientific Citation Text Generation](https://mlsanity.com/paper/2212.09577/) (2022)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction](https://mlsanity.com/paper/2012.11740/) (2020)
- [A Dataset on Malicious Paper Bidding in Peer Review](https://mlsanity.com/paper/2207.02303/) (2022)
- [Predicting the Citations of Scholarly Paper](https://mlsanity.com/paper/2008.05013/) (2020)
- [Predicting citation impact of research papers using GPT and other text embeddings](https://mlsanity.com/paper/2407.19942/) (2024)
- [MultiSChuBERT: Effective Multimodal Fusion for Scholarly Document Quality Prediction](https://mlsanity.com/paper/2308.07971/) (2023)
- [CiMaTe: Citation Count Prediction Effectively Leveraging the Main Text](https://mlsanity.com/paper/2410.04404/) (2024)

## Built on by (most cited first, 8 of 8)

- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Paraphrasing Adversarial Attack on LLM-as-a-Reviewer](https://mlsanity.com/paper/2601.06884/) (2026)
- [Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models](https://mlsanity.com/paper/2512.22145/) (2025)
- [Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning](https://mlsanity.com/paper/2512.02914/) (2025)
- [NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation](https://mlsanity.com/paper/2509.25179/) (2025)
- [AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents](https://mlsanity.com/paper/2602.14257/) (2026)
- [From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation](https://mlsanity.com/paper/2603.17588/) (2026)
- [GraphReview: Scientific Paper Evaluation via LLM-based Graph Evidence Expansion](https://mlsanity.com/paper/2605.27204/) (2026)

## Abstract (arXiv)

Foundation models are increasingly used in scientific research, but evaluating AI-generated scientific work remains challenging. While expert reviews are costly, large language models (LLMs) as proxy reviewers have proven to be unreliable. To address this, we investigate two automatic evaluation metrics, specifically citation count prediction and review score prediction. We parse all papers of OpenReview and augment each submission with its citation count, reference, and research hypothesis. Our findings reveal that citation count prediction is more viable than review score prediction, and predicting scores is more difficult purely from the research hypothesis than from the full paper. Furthermore, we show that a simple prediction model based solely on title and abstract outperforms LLM-based reviewers, though it still falls short of human-level consistency.