mlsanityData to 14 Oct 2026

ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?

arXiv 2309.08636 ↗First posted 14 Sep 2023cs.CL cs.AI cs.CYcited by 6

Not on the map. Why?

Abstract (arXiv)

Historical emphasis on writing mastery has shifted with advances in generative AI, especially in scientific writing. This study analysed six AI chatbots for scholarly writing in humanities and archaeology. Using methods that assessed factual correctness and scientific contribution, ChatGPT-4 showed the highest quantitative accuracy, closely followed by ChatGPT-3.5, Bing, and Bard. However, Claude 2 and Aria scored considerably lower. Qualitatively, all AIs exhibited proficiency in merging existing knowledge, but none produced original scientific content. Inter-estingly, our findings suggest ChatGPT-4 might represent a plateau in large language model size. This research emphasizes the unique, intricate nature of human research, suggesting that AI's emulation of human originality in scientific writing is challenging. As of 2023, while AI has transformed content generation, it struggles with original contributions in humanities. This may change as AI chatbots continue to evolve into LLM-powered software.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 27 of its 144 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

27 of 144 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2023A Survey of Large Language Modelscited by 2,576
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
15 more
  1. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  2. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  3. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  4. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  5. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  6. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 176
  7. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  8. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 101
  9. 2022X-Risk Analysis for AI Researchcited by 60
  10. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 57
  11. 2023ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishingcited by 52
  12. 2022"I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Datacited by 32
  13. 2022Current and Near-Term AI as a Potential Existential Risk Factorcited by 14
  14. 2016Back to the past: on the shoulders of an academic search engine giantcited by 3
  15. 2023ChatGPT cites the most-cited articles and journals, relying solely on Google Scholar's citation counts. As a result, AI may amplify the Matthew Effect in environmental sciencecited by 2

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 96
  2. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  3. 2024GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Educationcited by 14
  4. 2025Examining Linguistic Shifts in Academic Writing Before and After the Launch of ChatGPT: A Study on Preprint Paperscited by 6
  5. 2025Enhancing Abstractive Summarization of Scientific Papers Using Structure Informationcited by 2
  6. 2025A ChatGPT-based approach for questions generation in higher educationcited by 2