# Metag: A dataset to build agentic meta-reviewing capabilities

arXiv: https://arxiv.org/abs/2608.20488 · HTML: https://mlsanity.com/paper/2608.20488/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 14 of 14)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [Understanding Iterative Revision from Human-Written Text](https://mlsanity.com/paper/2203.03802/) (2022)
- [Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review](https://mlsanity.com/paper/2204.10805/) (2022)
- [ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews](https://mlsanity.com/paper/2306.12587/) (2023)
- [arXivEdits: Understanding the Human Revision Process in Scientific Writing](https://mlsanity.com/paper/2210.15067/) (2022)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond](https://mlsanity.com/paper/2205.11484/) (2022)
- [SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers](https://mlsanity.com/paper/2411.05338/) (2024)
- [Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation](https://mlsanity.com/paper/2305.14647/) (2023)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)

## Abstract (arXiv)

AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same time, the continuing growth in conference submissions has increased the burden on meta-reviewers, who must synthesize reviewer feedback, author rebuttals, and manuscript revisions. To address this concern, this paper introduces Metag, a dataset to accelerate the development of meta-reviewing agents, specifically to identify changes made to scientific articles during the review-rebuttal process. Each instance contains a reviewer concern, the author's proposed resolution, and the manuscript diffs implementing the stated change. Metag is collected by obtaining manuscript versions from before the review deadline and after acceptance, computing differences between the two documents, and asking human annotators to align these differences with action items from OpenReview discussions. The resulting dataset consists of 349 high-quality action items tied to paper differences and will enable building methods to empower meta reviewers to quickly identify whether authors have addressed reviewer statements and where in the paper those changes have been made, resulting in additional transparency and traceability throughout peer review. The dataset is publicly available at https://github.com/microsoft/Metag-dataset.