# Delving into LLM-assisted writing in biomedical publications through excess vocabulary

arXiv: https://arxiv.org/abs/2406.07016 · HTML: https://mlsanity.com/paper/2406.07016/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [The Science of Detecting LLM-Generated Texts](https://mlsanity.com/paper/2303.07205/) (2023)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization](https://mlsanity.com/paper/2402.13249/) (2024)
- [FABLES: Evaluating faithfulness and content selection in book-length summarization](https://mlsanity.com/paper/2404.01261/) (2024)
- [Machine Culture](https://mlsanity.com/paper/2311.11388/) (2023)
- [Measuring Implicit Bias in Explicitly Unbiased Large Language Models](https://mlsanity.com/paper/2402.04105/) (2024)
- [ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature](https://mlsanity.com/paper/2403.16887/) (2024)
- [Is ChatGPT Transforming Academics' Writing Style?](https://mlsanity.com/paper/2404.08627/) (2024)
- [Towards the relationship between AIGC in manuscript writing and author profiles: evidence from preprints in LLMs](https://mlsanity.com/paper/2404.15799/) (2024)
- [Detecting LLM-assisted writing in scientific communication: Are we there yet?](https://mlsanity.com/paper/2401.16807/) (2024)
- [Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in Arxiv Submissions using AI Detection Tool](https://mlsanity.com/paper/2403.13812/) (2024)

## Built on by (most cited first, 30 of 78)

- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions](https://mlsanity.com/paper/2411.05025/) (2024)
- [Scientific production in the era of Large Language Models](https://mlsanity.com/paper/2601.13187/) (2026)
- [The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models](https://mlsanity.com/paper/2502.11266/) (2025)
- [Diverse Preference Learning for Capabilities and Alignment](https://mlsanity.com/paper/2511.08594/) (2025)
- [The Impact of Large Language Models in Academia: from Writing to Speaking](https://mlsanity.com/paper/2409.13686/) (2024)
- [Does ChatGPT Have a Poetic Style?](https://mlsanity.com/paper/2410.15299/) (2024)
- [Academic journals' AI policies fail to curb the surge in AI-assisted academic writing](https://mlsanity.com/paper/2512.06705/) (2025)
- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [System 2 thinking in OpenAI's o1-preview model: Near-perfect performance on a mathematics exam](https://mlsanity.com/paper/2410.07114/) (2024)
- [AI use in American newspapers is widespread, uneven, and rarely disclosed](https://mlsanity.com/paper/2510.18774/) (2025)
- [Detecting LLM-Generated Text with Performance Guarantees](https://mlsanity.com/paper/2601.06586/) (2026)
- [Can Good Writing Be Generative? Expert-Level AI Writing Emerges through Fine-Tuning on High-Quality Books](https://mlsanity.com/paper/2601.18353/) (2026)
- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models](https://mlsanity.com/paper/2412.11385/) (2024)
- [Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy](https://mlsanity.com/paper/2406.17324/) (2024)
- [The Impact of AI-Generated Text on the Internet](https://mlsanity.com/paper/2604.26965/) (2026)
- [How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses](https://mlsanity.com/paper/2504.13038/) (2025)
- [Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity](https://mlsanity.com/paper/2509.22641/) (2025)
- [Multi-Novelty: Improve the Diversity and Novelty of Contents Generated by Large Language Models via inference-time Multi-Views Brainstorming](https://mlsanity.com/paper/2502.12700/) (2025)
- [NLLG Quarterly arXiv Report 09/24: What are the most influential current AI Papers?](https://mlsanity.com/paper/2412.12121/) (2024)
- [GPT Editors, Not Authors: The Stylistic Footprint of LLMs in Academic Preprints](https://mlsanity.com/paper/2505.17327/) (2025)
- [Research Novelty in Information Systems Journals After ChatGPT: Differences Across Institutional Language Contexts](https://mlsanity.com/paper/2603.22510/) (2026)
- [AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writing](https://mlsanity.com/paper/2605.25358/) (2026)
- [Invisible to humans, visible to machines: a preregistered audit of Unicode fidelity across four biomedical bibliographic APIs](https://mlsanity.com/paper/2606.24897/) (2026)
- [Towards Automating Scientific Review with Google's Paper Assistant Tool](https://mlsanity.com/paper/2606.28277/) (2026)
- [Most biomedical publications show signs of LLM-assisted writing](https://mlsanity.com/paper/2608.10715/) (2026)

## Abstract (arXiv)

Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. These models come with clear limitations: they can produce inaccurate information, reinforce existing biases, and be easily misused. Yet, many scientists use them for their scholarly writing. But how wide-spread is such LLM usage in the academic literature? To answer this question for the field of biomedical research, we present an unbiased, large-scale approach: we study vocabulary changes in over 15 million biomedical abstracts from 2010--2024 indexed by PubMed, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. This excess word analysis suggests that at least 13.5% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, reaching 40% for some subcorpora. We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the Covid pandemic.