mlsanityData to 14 Oct 2026

OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

arXiv 2412.11948 ↗First posted 16 Dec 2024cs.AIcited by 65

Not on the map. Why?

Abstract (arXiv)

We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer's recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

25 of 41 references.

References · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  6. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  7. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  8. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178
  9. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  10. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  11. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  12. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
13 more
  1. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 54
  2. 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 53
  3. 2022Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Reviewcited by 37
  4. 2017Design and Analysis of the NIPS 2016 Review Processcited by 33
  5. 2020What Can We Do to Improve Peer Review in NLP?cited by 32
  6. 2018Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratingscited by 28
  7. 2023Summarizing Multiple Documents with Conversational Structure for Meta-Review Generationcited by 27
  8. 2024Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publicationscited by 26
  9. 2019Simple and Effective Paraphrastic Similarity from Parallel Translationscited by 25
  10. 2019The Diversity-Innovation Paradox in Sciencecited by 21
  11. 2024LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Paperscited by 16
  12. 2020Uncovering Latent Biases in Text: Method and Application to Peer Reviewcited by 11
  13. 2024On the Detection of Reviewer-Author Collusion Rings From Paper Biddingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 43.

65 papers.

Sort:

By area:

  1. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  2. 2025Large language models for automated scholarly paper review: A surveycited by 61
  3. 2025Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussionscited by 23
  4. 2025OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientistscited by 18
  5. 2025ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Reviewcited by 16
  6. 2025How Far Are AI Scientists from Changing the World?cited by 15
  7. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 14
  8. 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 14
  9. 2025Prompt Injection Attacks on LLM Generated Reviews of Scientific Publicationscited by 13
  10. 2025Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedbackcited by 12
  11. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  12. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
38 more
  1. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  2. 2026MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibilitycited by 8
  3. 2026NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessmentcited by 7
  4. 2025When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injectioncited by 6
  5. 2025Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Modelscited by 6
  6. 2026DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoningcited by 6
  7. 2026RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mindcited by 6
  8. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  9. 2025NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimationcited by 4
  10. 2025A Human-AI Collaborative Workflow for Mathematical Discovery: A Case Study in Grover-Compatible Riemannian Optimizationcited by 4
  11. 2026ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Reviewcited by 4
  12. 2026GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responsescited by 4
  13. 2025Graph-Guided Passage Retrieval for Author-Centric Structured Feedbackcited by 3
  14. 2025SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systemscited by 3
  15. 2025ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentationcited by 3
  16. 2025Multi-Agent LLMs for Generating Research Limitationscited by 3
  17. 2026Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviewscited by 3
  18. 2026Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Communitycited by 3
  19. 2025Multimodal Peer Review Simulation with Actionable To-Do Recommendations for Community-Aware Manuscript Revisionscited by 2
  20. 2026What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Reviewcited by 2
  21. 2026No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisionscited by 2
  22. 2026RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Reviewcited by 2
  23. 2026Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generationcited by 2
  24. 2026Are We Truly Innovating? A Qualitative and Quantitative Study of Originality in AI Research Paperscited by 1
  25. 2026IntelliAsk: Learning to Ask High-Quality Research Questions via RLVRcited by 1
  26. 2026Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Paperscited by 1
  27. 2026Decoupling Scores and Text: The Politeness Principle in Peer Reviewcited by 1
  28. 2026Review Arcade: On the Human Alignment and Gameability of LLM Reviewscited by 1
  29. 2026Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generationcited by 1
  30. 2026LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challengescited by 1
  31. 2026Uncertainty-Aware Generation and Decision-Making Under Ambiguitycited by 1
  32. 2026Do Methods Support the Claims? Intra-Paper Verification for Peer Reviewcited by 1
  33. 2026How Closely Do LLM Reviews Align with Human Peer Review?cited by 1
  34. 2026ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchscited by 1
  35. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  36. 2025Few-Shot and Training-Free Review Generation via Conversational Promptingcited by 0
  37. 2025AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinementcited by 0
  38. 2026Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Rankingcited by 0

50 most cited of 65 All 65 (JSON)