mlsanityData to 7 Oct 2026

Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT

arXiv 2302.10198 ↗First posted 19 Feb 2023cs.CLcited by 155

Not on the map. Why?

Abstract (arXiv)

Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT's understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question-answering tasks. Additionally, by combining some advanced prompting strategies, we show that the understanding ability of ChatGPT can be further improved.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

31 of 44 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
19 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2022In-context Examples Selection for Machine Translationcited by 172
  13. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  14. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  15. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  16. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  17. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  18. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  19. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 71; 2024: 59; 2025: 17; 2026: 8.

155 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  4. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  5. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  6. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  8. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  9. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  10. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  11. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  12. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
38 more
  1. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  2. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  3. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  4. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  5. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  6. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  7. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  8. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  9. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  10. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  11. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  12. 2023Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detectioncited by 69
  13. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  14. 2024Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classificationcited by 66
  15. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 64
  16. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  17. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  18. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  19. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  20. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  21. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  22. 2023Towards Interpretable Mental Health Analysis with Large Language Modelscited by 53
  23. 2023Probing LLMs for hate speech detection: strengths and vulnerabilitiescited by 52
  24. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  25. 2023An Empirical Study on Information Extraction using Large Language Modelscited by 46
  26. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  27. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  28. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  29. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 41
  30. 2023Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understandingcited by 40
  31. 2023CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTractscited by 39
  32. 2023ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relationscited by 39
  33. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  34. 2024Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversationscited by 36
  35. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35
  36. 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 35
  37. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environmentscited by 35
  38. 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 33

50 most cited of 155 All 155 (JSON)