mlsanityData to 7 Oct 2026

ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Time

arXiv 2304.14106 ↗First posted 27 Apr 2023cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

ChatGPT has achieved great success and can be considered to have acquired an infrastructural status. There are abundant works for evaluating ChatGPT on benchmarks. However, existing benchmarks encounter two challenges: (1) Disregard for periodical evaluation and (2) Lack of fine-grained features. In this paper, we construct ChatLog, an ever-updating dataset with large-scale records of diverse long-form ChatGPT responses for 21 NLP benchmarks from March, 2023 to now. We conduct a comprehensive performance evaluation to find that most capabilities of ChatGPT improve over time except for some abilities, and there exists a step-wise evolving pattern of ChatGPT. We further analyze the inherent characteristics of ChatGPT by extracting the knowledge and linguistic features. We find some stable features that stay unchanged and apply them on the detection of ChatGPT-generated texts to improve the robustness of cross-version detection. We will continuously maintain our project at urlhttps://github.com/THU-KEG/ChatLog/.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

33 of 55 references.

References · 33

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  11. 2019ELI5: Long Form Question Answeringcited by 567
  12. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 490
21 more
  1. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  2. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  3. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  4. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  5. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  6. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  7. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  8. 2022Unified Structure Generation for Universal Information Extractioncited by 220
  9. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  10. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  11. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  12. 2023Can we trust the evaluation on ChatGPT?cited by 63
  13. 2023Will ChatGPT get you caught? Rethinking of Plagiarism Detectioncited by 41
  14. 2023ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT modelscited by 41
  15. 2021Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Featurescited by 36
  16. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  17. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  18. 2023AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essayscited by 9
  19. 2023Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Datacited by 7
  20. 2023Comparative Analysis of CHATGPT and the evolution of language modelscited by 6
  21. 2017Dynamic Fusion Networks for Machine Reading Comprehensioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 3; 2025: 1; 2026: 1.

12 papers.

  1. 2023How is ChatGPT's behavior changing over time?cited by 265
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  3. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  4. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  5. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  6. 2024AI for social science and social science of AI: A Surveycited by 29
  7. 2024Measuring Social Norms of Large Language Modelscited by 16
  8. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Modelscited by 12
  9. 2025Patterns Over Principles: The Fragility of Inductive Reasoning in LLMs under Noisy Observationscited by 12
  10. 2023Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queriescited by 4
  11. 2024Enhancing literature review with LLM and NLP methods. Algorithmic trading casecited by 2
  12. 2026Large Databases Need Small, Open-Weight Language Modelscited by 1