mlsanityData to 7 Oct 2026

Automated scholarly paper review: Concepts, technologies, and challenges

arXiv 2111.07533 ↗First posted 15 Nov 2021cs.AI cs.CL cs.DLcited by 21

Not on the map. Why?

Abstract (arXiv)

Peer review is a widely accepted mechanism for research evaluation, playing a pivotal role in academic publishing. However, criticisms have long been leveled at this mechanism, mostly because of its poor efficiency and low reproducibility. Recent years have seen the application of artificial intelligence (AI) in assisting the peer review process. Nonetheless, with the involvement of humans, such limitations remain inevitable. In this paper, we propose the concept and pipeline of automated scholarly paper review (ASPR) and review the relevant literature and technologies of achieving a full-scale computerized review process. On the basis of the review and discussion, we conclude that there is already corresponding research and preliminary implementation at each stage of ASPR. We further look into the challenges in ASPR with the existing technologies. The major difficulties lie in inadequate data, imperfect document parsing and representation, defective humanunicodex2013computer interaction, and flawed deep logical reasoning. Moreover, we point out the future directions and discuss the possible moral and ethical issues of ASPR. In the foreseeable future, ASPR and peer review will coexist in a reinforcing manner before ASPR is able to fully undertake the reviewing workload from humans.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 55 of its 198 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

55 of 198 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  10. 2014Distributed Representations of Sentences and Documentscited by 677
  11. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  12. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
38 more
  1. 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
  2. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  3. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 135
  4. 2019The Ethics of AI Ethics -- An Evaluation of Guidelinescited by 129
  5. 2019Artificial Intelligence: the global landscape of ethics guidelinescited by 107
  6. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  7. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  8. 2019ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networkscited by 73
  9. 2019DocBERT: BERT for Document Classificationcited by 68
  10. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 64
  11. 2021Can We Automate Scientific Reviewing?cited by 63
  12. 2019Linguistic generalization and compositionality in modern artificial neural networkscited by 58
  13. 2021MathBERT: A Pre-Trained Model for Mathematical Formula Understandingcited by 58
  14. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 53
  15. 2019Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Constructioncited by 50
  16. 2020AxCell: Automatic Extraction of Results from Machine Learning Paperscited by 48
  17. 2019Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrievalcited by 37
  18. 2021Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documentscited by 33
  19. 2019Does My Rebuttal Matter? Insights from a Major NLP Conferencecited by 31
  20. 2021TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metricscited by 29
  21. 2019TalkSumm: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talkscited by 20
  22. 2016Compositional Sequence Labeling Models for Error Detection in Learner Writingcited by 16
  23. 2014A Corpus of Sentence-level Revisions in Academic Writing: A Step towards Understanding Statement Strength in Communicationcited by 14
  24. 2019Context is Key: Grammatical Error Detection with Contextual Word Representationscited by 14
  25. 2019PaRe: A Paper-Reviewer Matching Approach Using a Common Topic Spacecited by 14
  26. 2019Generating Long and Informative Reviews with Aspect-Aware Coarse-to-Fine Decodingcited by 13
  27. 2020SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Predictioncited by 9
  28. 2018TabVec: Table Vectors for Classification of Web Tablescited by 8
  29. 2018Automatic Academic Paper Rating Based on Modularized Hierarchical Convolutional Neural Networkcited by 7
  30. 2019A Joint Model for Multimodal Document Quality Assessmentcited by 7
  31. 2018Deep Paper Gestaltcited by 6
  32. 2015Comparing the writing style of real and artificial paperscited by 5
  33. 2019Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcomecited by 5
  34. 2018TAP-DLND 1.0 : A Corpus for Document Level Novelty Detectioncited by 4
  35. 2021COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviewscited by 4
  36. 2019An Edit-centric Approach for Wikipedia Article Quality Assessmentcited by 3
  37. 2016Sentence-Level Grammatical Error Identification as Sequence-to-Sequence Correctioncited by 2
  38. 2017Artificial intelligence in peer review: How can evolutionary computation support journal editors?cited by 2

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 9; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025Large language models for automated scholarly paper review: A surveycited by 59
  2. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  3. 2022MOPRD: A multidisciplinary open peer review datasetcited by 32
  4. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  5. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12
  6. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  7. 2025Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Papercited by 10
  8. 2025Insights from the ICLR Peer Review and Rebuttal Processcited by 8
  9. 2026OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessmentcited by 4
  10. 2023Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Reviewcited by 3
  11. 2024Streamlining the review process: AI-generated annotations in research manuscriptscited by 2
  12. 2026REM-CTX: Automated Peer Review via Reinforcement Learning with Auxiliary Contextcited by 2
9 more
  1. 2026Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNscited by 1
  2. 2026What happens when reviewers receive AI feedback in their reviews?cited by 1
  3. 2026FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Researchcited by 1
  4. 2025PEMUTA: Pedagogically-Enriched Multi-Granular Undergraduate Thesis Assessmentcited by 0
  5. 2025LLMs Can Assist with Proposal Selection at Large User Facilitiescited by 0
  6. 2026Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Paperscited by 0
  7. 2026Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AIcited by 0
  8. 2026Aspect-Based Sentiment Evolution and its Correlation with Review Rounds in Multi-Round Peer Reviews: A Deep Learning Approachcited by 0
  9. 2026InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttalcited by 0