# How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks

arXiv: https://arxiv.org/abs/2303.00293 · HTML: https://mlsanity.com/paper/2303.00293/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)

## Built on by (most cited first, 30 of 58)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages](https://mlsanity.com/paper/2402.10753/) (2024)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family](https://mlsanity.com/paper/2303.07992/) (2023)
- [SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models](https://mlsanity.com/paper/2403.02509/) (2024)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning](https://mlsanity.com/paper/2401.08326/) (2024)
- [ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models](https://mlsanity.com/paper/2304.07666/) (2023)
- [Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models](https://mlsanity.com/paper/2307.10236/) (2023)
- [GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP](https://mlsanity.com/paper/2305.14976/) (2023)
- [ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios](https://mlsanity.com/paper/2401.00741/) (2024)
- [Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Tasks](https://mlsanity.com/paper/2305.08714/) (2023)
- [Evaluating General-Purpose AI with Psychometrics](https://mlsanity.com/paper/2310.16379/) (2023)
- [Robust Prompt Optimization for Large Language Models Against Distribution Shifts](https://mlsanity.com/paper/2305.13954/) (2023)
- [LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems](https://mlsanity.com/paper/2312.09348/) (2023)
- [Counter Turing Test CT^2: AI-Generated Text Detection is Not as Easy as You May Think -- Introducing AI Detectability Index](https://mlsanity.com/paper/2310.05030/) (2023)
- [Robots in the Middle: Evaluating LLMs in Dispute Resolution](https://mlsanity.com/paper/2410.07053/) (2024)
- [LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2402.14568/) (2024)
- [TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use](https://mlsanity.com/paper/2412.15495/) (2024)
- [Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2406.11890/) (2024)
- [On Sarcasm Detection with OpenAI GPT-based Models](https://mlsanity.com/paper/2312.04642/) (2023)
- [Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation](https://mlsanity.com/paper/2305.13067/) (2023)
- [Boosting Data Analytics With Synthetic Volume Expansion](https://mlsanity.com/paper/2310.17848/) (2023)
- [Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels](https://mlsanity.com/paper/2509.16596/) (2025)
- [LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning](https://mlsanity.com/paper/2412.13626/) (2024)

## Abstract (arXiv)

The GPT-3.5 models have demonstrated impressive performance in various Natural Language Processing (NLP) tasks, showcasing their strong understanding and reasoning capabilities. However, their robustness and abilities to handle various complexities of the open world have yet to be explored, which is especially crucial in assessing the stability of models and is a key aspect of trustworthy AI. In this study, we perform a comprehensive experimental analysis of GPT-3.5, exploring its robustness using 21 datasets (about 116K test samples) with 66 text transformations from TextFlint that cover 9 popular Natural Language Understanding (NLU) tasks. Our findings indicate that while GPT-3.5 outperforms existing fine-tuned models on some tasks, it still encounters significant robustness degradation, such as its average performance dropping by up to 35.74% and 43.59% in natural language inference and sentiment analysis tasks, respectively. We also show that GPT-3.5 faces some specific robustness challenges, including robustness instability, prompt sensitivity, and number sensitivity. These insights are valuable for understanding its limitations and guiding future research in addressing these challenges to enhance GPT-3.5's overall performance and generalization abilities.