# GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science

arXiv: https://arxiv.org/abs/2312.03769 · HTML: https://mlsanity.com/paper/2312.03769/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023)
- [Radiology-GPT: A Large Language Model for Radiology](https://mlsanity.com/paper/2306.08666/) (2023)
- [AI-assisted coding: Experiments with GPT-4](https://mlsanity.com/paper/2304.13187/) (2023)
- [GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT](https://mlsanity.com/paper/2307.07930/) (2023)
- [Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding](https://mlsanity.com/paper/2304.05368/) (2023)
- [DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule Graphs](https://mlsanity.com/paper/2309.03907/) (2023)
- [AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiology](https://mlsanity.com/paper/2306.10095/) (2023)
- [ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback](https://mlsanity.com/paper/2305.18090/) (2023)
- [Can ChatGPT be used to generate scientific hypotheses?](https://mlsanity.com/paper/2304.12208/) (2023)
- [CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study](https://mlsanity.com/paper/2307.11346/) (2023)
- [ChatMOF: An Autonomous AI System for Predicting and Generating Metal-Organic Frameworks](https://mlsanity.com/paper/2308.01423/) (2023)
- [An Interdisciplinary Outlook on Large Language Models for Scientific Research](https://mlsanity.com/paper/2311.04929/) (2023)
- [CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models](https://mlsanity.com/paper/2304.10946/) (2023)

## Built on by (most cited first, 5 of 5)

- [DeepSeek vs. ChatGPT vs. Claude: A Comparative Study for Scientific Computing and Scientific Machine Learning Tasks](https://mlsanity.com/paper/2502.17764/) (2025)
- [ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2512.03476/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [FMEnets: Flow, Material, and Energy networks for non-ideal plug flow reactor design](https://mlsanity.com/paper/2505.20300/) (2025)
- [Hidden State Poisoning Attacks against Mamba-based Language Models](https://mlsanity.com/paper/2601.01972/) (2026)

## Abstract (arXiv)

The new polymath Large Language Models (LLMs) can speed-up greatly scientific reviews, possibly using more unbiased quantitative metrics, facilitating cross-disciplinary connections, and identifying emerging trends and research gaps by analyzing large volumes of data. However, at the present time, they lack the required deep understanding of complex methodologies, they have difficulty in evaluating innovative claims, and they are unable to assess ethical issues and conflicts of interest. Herein, we consider 13 GPT-related papers across different scientific domains, reviewed by a human reviewer and SciSpace, a large language model, with the reviews evaluated by three distinct types of evaluators, namely GPT-3.5, a crowd panel, and GPT-4. We found that 50% of SciSpace's responses to objective questions align with those of a human reviewer, with GPT-4 (informed evaluator) often rating the human reviewer higher in accuracy, and SciSpace higher in structure, clarity, and completeness. In subjective questions, the uninformed evaluators (GPT-3.5 and crowd panel) showed varying preferences between SciSpace and human responses, with the crowd panel showing a preference for the human responses. However, GPT-4 rated them equally in accuracy and structure but favored SciSpace for completeness.