# AI vs. Human -- Differentiation Analysis of Scientific Content Generation

arXiv: https://arxiv.org/abs/2301.10416 · HTML: https://mlsanity.com/paper/2301.10416/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 59 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021)
- [ChatGPT: The End of Online Exam Integrity?](https://mlsanity.com/paper/2212.09292/) (2022)
- [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [DeepFake Detection: Current Challenges and Next Steps](https://mlsanity.com/paper/2003.09234/) (2020)
- [Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models](https://mlsanity.com/paper/2209.06506/) (2022)
- [RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text](https://mlsanity.com/paper/2010.03070/) (2020)
- [CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation](https://mlsanity.com/paper/2204.00862/) (2022)
- [Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts](https://mlsanity.com/paper/1808.06161/) (2018)
- [IGA : An Intent-Guided Authoring Assistant](https://mlsanity.com/paper/2104.07000/) (2021)
- [SynSciPass: detecting appropriate uses of scientific text generation](https://mlsanity.com/paper/2209.03742/) (2022)

## Built on by (most cited first, 21 of 21)

- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Testing of Detection Tools for AI-Generated Text](https://mlsanity.com/paper/2306.15666/) (2023)
- [People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text](https://mlsanity.com/paper/2501.15654/) (2025)
- [Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis](https://mlsanity.com/paper/2304.05534/) (2023)
- [Are AI-Generated Text Detectors Robust to Adversarial Perturbations?](https://mlsanity.com/paper/2406.01179/) (2024)
- [CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings](https://mlsanity.com/paper/2503.13733/) (2025)
- [Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Texts](https://mlsanity.com/paper/2410.14677/) (2024)
- [AI-Generated Text Detection and Classification Based on BERT Deep Learning Algorithm](https://mlsanity.com/paper/2405.16422/) (2024)
- [Examining Linguistic Shifts in Academic Writing Before and After the Launch of ChatGPT: A Study on Preprint Papers](https://mlsanity.com/paper/2505.12218/) (2025)
- [Linguistic and Embedding-Based Profiling of Texts generated by Humans and Large Language Models](https://mlsanity.com/paper/2507.13614/) (2025)
- [Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communication](https://mlsanity.com/paper/2502.08896/) (2025)
- [DeepPavlov at SemEval-2024 Task 8: Leveraging Transfer Learning for Detecting Boundaries of Machine-Generated Texts](https://mlsanity.com/paper/2405.10629/) (2024)
- [Detection of Fake Generated Scientific Abstracts](https://mlsanity.com/paper/2304.06148/) (2023)
- [Distinguishing Chatbot from Human](https://mlsanity.com/paper/2408.04647/) (2024)
- [Advacheck at GenAI Detection Task 1: AI Detection Powered by Domain-Aware Multi-Tasking](https://mlsanity.com/paper/2411.11736/) (2024)
- [Decoding AI and Human Authorship: Nuances Revealed Through NLP and Statistical Analysis](https://mlsanity.com/paper/2408.00769/) (2024)
- [Enhancing Authorship Attribution through Embedding Fusion: A Novel Approach with Masked and Encoder-Decoder Language Models](https://mlsanity.com/paper/2411.00411/) (2024)
- [PhDGPT: Introducing a psychometric and linguistic dataset about how large language models perceive graduate students and professors in psychology](https://mlsanity.com/paper/2411.10473/) (2024)
- [Feature Extraction and Analysis for GPT-Generated Text](https://mlsanity.com/paper/2503.13687/) (2025)
- [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026)
- [What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience](https://mlsanity.com/paper/2605.19936/) (2026)

## Abstract (arXiv)

Recent neural language models have taken a significant step forward in producing remarkably controllable, fluent, and grammatical text. Although studies have found that AI-generated text is not distinguishable from human-written text for crowd-sourcing workers, there still exist errors in AI-generated text which are even subtler and harder to spot. We primarily focus on the scenario in which scientific AI writing assistant is deeply involved. First, we construct a feature description framework to distinguish between AI-generated text and human-written text from syntax, semantics, and pragmatics based on the human evaluation. Then we utilize the features, i.e., writing style, coherence, consistency, and argument logistics, from the proposed framework to analyze two types of content. Finally, we adopt several publicly available methods to investigate the gap of between AI-generated scientific text and human-written scientific text by AI-generated scientific text detection models. The results suggest that while AI has the potential to generate scientific content that is as accurate as human-written content, there is still a gap in terms of depth and overall quality. The AI-generated scientific content is more likely to contain errors in factual issues. We find that there exists a "writing style" gap between AI-generated scientific text and human-written scientific text. Based on the analysis result, we summarize a series of model-agnostic and distribution-agnostic features for detection tasks in other domains. Findings in this paper contribute to guiding the optimization of AI models to produce high-quality content and addressing related ethical and security concerns.