mlsanityData to 14 Oct 2026

A Multi-Task Evaluation of LLMs' Processing of Academic Text Input

arXiv 2508.11779 ↗First posted 15 Aug 2025cs.CL econ.GN q-fin.ECcited by 5

Not on the map. Why?

Abstract (arXiv)

How much large language models (LLMs) can aid scientific discovery, notably in assisting academic peer review, is in heated debate. Between a literature digest and a human-comparable research assistant lies their practical application potential. We organize individual tasks that computer science studies employ in separate terms into a guided and robust workflow to evaluate LLMs' processing of academic text input. We employ four tasks in the assessment: content reproduction/comparison/scoring/reflection, each demanding a specific role of the LLM (oracle/judgmental arbiter/knowledgeable arbiter/collaborator) in assisting scholarly works, and altogether testing LLMs with questions that increasingly require intellectual capabilities towards a solid understanding of scientific texts to yield desirable solutions. We exemplify a rigorous performance evaluation with detailed instructions on the prompts. Adopting first-rate Information Systems articles at three top journals as the input texts and an abundant set of text metrics, we record a compromised performance of the leading LLM - Google's Gemini: its summary and paraphrase of academic text is acceptably reliable; using it to rank texts through pairwise text comparison is faintly scalable; asking it to grade academic texts is prone to poor discrimination; its qualitative reflection on the text is self-consistent yet hardly insightful to inspire meaningful research. This evidence against an endorsement of LLMs' text-processing capabilities is consistent across metric-based internal (linguistic assessment), external (comparing to the ground truth), and human evaluation, and is robust to the variations of the prompt. Overall, we do not recommend an unchecked use of LLMs in constructing peer reviews.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 57 of its 174 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

57 of 174 references.

References · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  8. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  9. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  10. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  11. 2023Mathematical Capabilities of ChatGPTcited by 268
  12. 2024Evolutionary Optimization of Model Merging Recipescited by 208
38 more
  1. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  2. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  3. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  4. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  5. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  6. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  7. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  8. 2018Discovering physical concepts with neural networkscited by 117
  9. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  10. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  11. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  12. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  13. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  14. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  15. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  16. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 70
  17. 2022On scientific understanding with artificial intelligencecited by 70
  18. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  19. 2021Can We Automate Scientific Reviewing?cited by 63
  20. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  21. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 55
  22. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 54
  23. 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformationcited by 51
  24. 2023GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Studycited by 40
  25. 2023Bias of AI-Generated Content: An Examination of News Produced by Large Language Modelscited by 38
  26. 2024The Value, Benefits, and Concerns of Generative AI-Powered Assistance in Writingcited by 37
  27. 2024Automated Focused Feedback Generation for Scientific Writing Assistancecited by 33
  28. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  29. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 32
  30. 2024Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domainscited by 30
  31. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  32. 2023Accelerating science with human-aware artificial intelligencecited by 28
  33. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  34. 2015A review of the literature on citation impact indicatorscited by 14
  35. 2023Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluationcited by 14
  36. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12
  37. 2024ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewingcited by 12
  38. 2024How Does the Disclosure of AI Assistance Affect the Perceptions of Writing?cited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendationcited by 1
  2. 2026Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judgescited by 1
  3. 2026Interpretable Markov-Based Spatiotemporal Risk Surfaces for Missing-Child Search Planning with Reinforcement Learning and LLM-Based Quality Assurancecited by 0
  4. 2026A Consensus-Driven Multi-LLM Pipeline for Missing-Person Investigationscited by 0
  5. 2026LLM-based Schema-Guided Extraction and Validation of Missing-Person Intelligence from Heterogeneous Data Sourcescited by 0