# MetaSyn: A Benchmark for LLM Agents on Meta-Analysis Articles from Nature Portfolio

arXiv: https://arxiv.org/abs/2606.17041 · HTML: https://mlsanity.com/paper/2606.17041/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 22 of 22)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles](https://mlsanity.com/paper/2010.14235/) (2020)
- [Multi-document Summarization via Deep Learning Techniques: A Survey](https://mlsanity.com/paper/2011.04843/) (2020)
- [SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing](https://mlsanity.com/paper/2503.04629/) (2025)
- [Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages](https://mlsanity.com/paper/2310.17526/) (2023)
- [Open Source Software for Efficient and Transparent Reviews](https://mlsanity.com/paper/2006.12166/) (2020)
- [RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects](https://mlsanity.com/paper/2501.18365/) (2025)
- [Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.00175/) (2024)
- [Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts](https://mlsanity.com/paper/2308.10410/) (2023)
- [Instruct Large Language Models to Generate Scientific Literature Survey Step by Step](https://mlsanity.com/paper/2408.07884/) (2024)
- [CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews](https://mlsanity.com/paper/2311.12474/) (2023)

## Abstract (arXiv)

Systematic review and meta-analysis is an important method for scientific research. It comprehensively studies target research questions by combining evidence from multiple independent studies following a standard and rigid protocol (i.e., PI/ECO). This is valuable for facilitating reliable and sustainable research across multiple domains, including but not limited to physics, chemistry, psychology, and medical science. Traditional meta-analysis is both mind-intensive and labor-intensive, as it requires professionals to search, screen, and synthesize tens of thousands of studies, which usually takes months or even years. Recent advances in AI, particularly LLM agents, have the potential to significantly automate this process, but whether they can conduct reliable meta-analysis is still unknown. To this end, we introduce MetaSyn, a dataset and evaluation protocol built from 422 expert-curated meta-analyses manually selected from more than 34,000 published articles in Nature Portfolio journals. We collected the research questions with structured eligibility criteria, the studies included by the original reviewers, and a shared PubMed-anchored corpus containing both eligible studies and plausible but ineligible distractors to build a comprehensive benchmark for meta-analysis. Our experiments on multiple LLMs and baseline methods show that existing AI systems are far from perfect. We also conducted stage-wise evaluation and analysis to shed light on why existing AI systems fall short on meta-analysis.