mlsanityData to 7 Oct 2026

How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks

arXiv 2303.00293 ↗First posted 1 Mar 2023cs.CLcited by 58

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The GPT-3.5 models have demonstrated impressive performance in various Natural Language Processing (NLP) tasks, showcasing their strong understanding and reasoning capabilities. However, their robustness and abilities to handle various complexities of the open world have yet to be explored, which is especially crucial in assessing the stability of models and is a key aspect of trustworthy AI. In this study, we perform a comprehensive experimental analysis of GPT-3.5, exploring its robustness using 21 datasets (about 116K test samples) with 66 text transformations from TextFlint that cover 9 popular Natural Language Understanding (NLU) tasks. Our findings indicate that while GPT-3.5 outperforms existing fine-tuned models on some tasks, it still encounters significant robustness degradation, such as its average performance dropping by up to 35.74% and 43.59% in natural language inference and sentiment analysis tasks, respectively. We also show that GPT-3.5 faces some specific robustness challenges, including robustness instability, prompt sensitivity, and number sensitivity. These insights are valuable for understanding its limitations and guiding future research in addressing these challenges to enhance GPT-3.5's overall performance and generalization abilities.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 40 of its 57 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

40 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
27 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2018Deep contextualized word representationscited by 2,285
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  12. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  13. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  14. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  15. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  16. 2023Mathematical Capabilities of ChatGPTcited by 263
  17. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  18. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  19. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  20. 2021Trustworthy AI: A Computational Perspectivecited by 93
  21. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  22. 2019Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networkscited by 62
  23. 2018Universal Language Model Fine-tuning for Text Classificationcited by 50
  24. 2019Attentional Encoder Network for Targeted Sentiment Classificationcited by 48
  25. 2021Learning from Noisy Labels for Entity-Centric Information Extractioncited by 34
  26. 2019GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognitioncited by 18
  27. 2019A Hybrid Neural Network Model for Commonsense Reasoningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 20; 2025: 12; 2026: 4.

58 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  5. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  6. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 167
  7. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  8. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  9. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  10. 2024SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Modelscited by 54
  11. 2023Towards Interpretable Mental Health Analysis with Large Language Modelscited by 53
  12. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
38 more
  1. 2023ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT modelscited by 41
  2. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  3. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  4. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  5. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25
  6. 2023Evaluating General-Purpose AI with Psychometricscited by 23
  7. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  8. 2023LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systemscited by 17
  9. 2023Counter Turing Test CT^2: AI-Generated Text Detection is Not as Easy as You May Think -- Introducing AI Detectability Indexcited by 14
  10. 2024Robots in the Middle: Evaluating LLMs in Dispute Resolutioncited by 13
  11. 2024LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognitioncited by 11
  12. 2024TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Usecited by 11
  13. 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9
  14. 2023Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentationcited by 7
  15. 2023Boosting Data Analytics With Synthetic Volume Expansioncited by 7
  16. 2023On Sarcasm Detection with OpenAI GPT-based Modelscited by 7
  17. 2025Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levelscited by 7
  18. 2024LIFT: Improving Long Context Understanding Through Long Input Fine-Tuningcited by 6
  19. 2025MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extractioncited by 6
  20. 2023Knowledge Graph Question Answering for Materials Science (KGQA4MAT): Developing Natural Language Interface for Metal-Organic Frameworks Knowledge Graph (MOF-KG) Using LLMcited by 5
  21. 2024Comparative Study of Domain Driven Terms Extraction Using Large Language Modelscited by 5
  22. 2024PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimizationcited by 5
  23. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  24. 2024Motion Generation from Fine-grained Textual Descriptionscited by 4
  25. 2025Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacementscited by 4
  26. 2024Continual Few-shot Event Detection via Hierarchical Augmentation Networkscited by 3
  27. 2025SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Spacecited by 3
  28. 2025Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matchingcited by 3
  29. 2023Rethinking STS and NLI in Large Language Modelscited by 2
  30. 2024Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challengescited by 2
  31. 2024RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructionscited by 2
  32. 202460 Data Points are Sufficient to Fine-Tune LLMs for Question-Answeringcited by 2
  33. 2025Enhancing NER Performance in Low-Resource Pakistani Languages using Cross-Lingual Data Augmentationcited by 2
  34. 2024Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signalscited by 1
  35. 2025Pastiche Novel Generation Creating: Fan Fiction You Love in Your Favorite Author's Stylecited by 1
  36. 2025Beyond Words: Infusing Conversational Agents with Human-like Typing Behaviorscited by 1
  37. 2023Language models as master equation solverscited by 0
  38. 2024Counter Turing Test (CT2): Investigating AI-Generated Text Detection for Hindi -- Ranking LLMs based on Hindi AI Detectability Index (ADIhi)cited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Prompting GPT-3 To Be Reliable
  2. 2023Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording
  3. 2025Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
  4. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts
  5. 2025Benchmarking Reasoning Robustness in Large Language Models
  6. 2026Robust Reasoning Benchmark
  7. 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets
  8. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond