# ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Time

arXiv: https://arxiv.org/abs/2304.14106 · HTML: https://mlsanity.com/paper/2304.14106/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Can we trust the evaluation on ChatGPT?](https://mlsanity.com/paper/2303.12767/) (2023)
- [ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models](https://mlsanity.com/paper/2304.07666/) (2023)
- [Will ChatGPT get you caught? Rethinking of Plagiarism Detection](https://mlsanity.com/paper/2302.04335/) (2023)
- [Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features](https://mlsanity.com/paper/2109.12258/) (2021)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation](https://mlsanity.com/paper/2204.00862/) (2022)
- [AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays](https://mlsanity.com/paper/2304.14276/) (2023)

## Built on by (most cited first, 12 of 12)

- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [AI for social science and social science of AI: A Survey](https://mlsanity.com/paper/2401.11839/) (2024)
- [Measuring Social Norms of Large Language Models](https://mlsanity.com/paper/2404.02491/) (2024)
- [FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models](https://mlsanity.com/paper/2311.09829/) (2023)
- [Patterns Over Principles: The Fragility of Inductive Reasoning in LLMs under Noisy Observations](https://mlsanity.com/paper/2502.16169/) (2025)
- [Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries](https://mlsanity.com/paper/2308.11189/) (2023)
- [Enhancing literature review with LLM and NLP methods. Algorithmic trading case](https://mlsanity.com/paper/2411.05013/) (2024)
- [Large Databases Need Small, Open-Weight Language Models](https://mlsanity.com/paper/2606.31808/) (2026)

## Abstract (arXiv)

ChatGPT has achieved great success and can be considered to have acquired an infrastructural status. There are abundant works for evaluating ChatGPT on benchmarks. However, existing benchmarks encounter two challenges: (1) Disregard for periodical evaluation and (2) Lack of fine-grained features. In this paper, we construct ChatLog, an ever-updating dataset with large-scale records of diverse long-form ChatGPT responses for 21 NLP benchmarks from March, 2023 to now. We conduct a comprehensive performance evaluation to find that most capabilities of ChatGPT improve over time except for some abilities, and there exists a step-wise evolving pattern of ChatGPT. We further analyze the inherent characteristics of ChatGPT by extracting the knowledge and linguistic features. We find some stable features that stay unchanged and apply them on the detection of ChatGPT-generated texts to improve the robustness of cross-version detection. We will continuously maintain our project at urlhttps://github.com/THU-KEG/ChatLog/.