mlsanityData to 7 Oct 2026

Differentiate ChatGPT-generated and Human-written Medical Texts

arXiv 2304.11567 ↗First posted 23 Apr 2023cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Background: Large language models such as ChatGPT are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the Internet. However, medical texts such as clinical notes and diagnoses require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to healthcare and the general public. Objective: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT, and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first construct a suite of datasets containing medical texts written by human experts and generated by ChatGPT. In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally, we design and implement machine learning methods to detect medical text generated by ChatGPT. Results: Medical texts written by humans are more concrete, more diverse, and typically contain more useful information, while medical texts generated by ChatGPT pay more attention to fluency and logic, and usually express general terminologies rather than effective information specific to the context of the problem. A BERT-based model can effectively detect medical texts generated by ChatGPT, and the F1 exceeds 95%.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 29 of its 61 references that are in the map. First posted on arXiv 23 Apr 2023.

Builds on

29 of 61 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
17 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  2. 2019Publicly Available Clinical BERT Embeddingscited by 995
  3. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  4. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  5. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  6. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  7. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  8. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  9. 2023ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Textcited by 84
  10. 2023An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPTcited by 57
  11. 2023When Brain-inspired AI Meets AGIcited by 54
  12. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  13. 2022Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approachescited by 39
  14. 2023Context Matters: A Strategy to Pre-train Language Model for Science Educationcited by 32
  15. 2023Applying BERT and ChatGPT for Sentiment Analysis of Lyme Disease in Scientific Literaturecited by 26
  16. 2023Mask-guided BERT for Few Shot Text Classificationcited by 25
  17. 2022Coarse-to-fine Knowledge Graph Domain Adaptation based on Distantly-supervised Iterative Trainingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 7; 2025: 4; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  2. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  3. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  4. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  5. 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
  6. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  7. 2023AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiologycited by 34
  8. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
  9. 2023Segment Anything Model (SAM) for Radiation Oncologycited by 24
  10. 2025AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guaranteescited by 20
  11. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  12. 2023RadOnc-GPT: A Large Language Model for Radiation Oncologycited by 18
14 more
  1. 2024AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italiancited by 12
  2. 2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
  3. 2023Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated Textcited by 10
  4. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 10
  5. 2024Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomycited by 7
  6. 2025Linguistic Characteristics of AI-Generated Text: A Surveycited by 7
  7. 2023Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classificationcited by 5
  8. 2024Differentiating Between Human-Written and AI-Generated Texts Using Automatically Extracted Linguistic Featurescited by 5
  9. 2023Evaluating multiple large language models in pediatric ophthalmologycited by 4
  10. 2025Large Language Models for Bioinformaticscited by 3
  11. 2023From Military to Healthcare: Adopting and Expanding Ethical Principles for Generative Artificial Intelligencecited by 2
  12. 2026ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injectioncited by 2
  13. 2025How Do LLM-Generated Texts Impact Term-Based Retrieval Models?cited by 0
  14. 2026What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?cited by 0