mlsanityData to 7 Oct 2026

On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

arXiv 2302.12095 ↗First posted 22 Feb 2023cs.AI cs.CL cs.LGcited by 208

Not on the map. Why?

Abstract (arXiv)

ChatGPT is a recent chatbot service released by OpenAI and is receiving increasing attention over the past few months. While evaluations of various aspects of ChatGPT have been done, its robustness, i.e., the performance to unexpected inputs, is still unclear to the public. Robustness is of particular concern in responsible AI, especially for safety-critical applications. In this paper, we conduct a thorough evaluation of the robustness of ChatGPT from the adversarial and out-of-distribution (OOD) perspective. To do so, we employ the AdvGLUE and ANLI benchmarks to assess adversarial robustness and the Flipkart review and DDXPlus medical diagnosis datasets for OOD evaluation. We select several popular foundation models as baselines. Results show that ChatGPT shows consistent advantages on most adversarial and OOD classification and translation tasks. However, the absolute performance is far from perfection, which suggests that adversarial and OOD robustness remains a significant threat to foundation models. Moreover, ChatGPT shows astounding performance in understanding dialogue-related texts and we find that it tends to provide informal suggestions for medical tasks instead of definitive answers. Finally, we present in-depth discussions of possible research directions.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Builds on 50 of its 75 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

50 of 75 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  13. 2021Scaling Vision Transformerscited by 993
  14. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  15. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  16. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  17. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  18. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  19. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  20. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  21. 2018Recognition in Terra Incognitacited by 655
  22. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  23. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  24. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  25. 2023MusicLM: Generating Music From Textcited by 442
  26. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  27. 2021Deep Long-Tailed Learning: A Surveycited by 416
  28. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  29. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  30. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  31. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  32. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  33. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  34. 2022DDXPlus: A New Dataset For Automatic Medical Diagnosiscited by 87
  35. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
  36. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  37. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  38. 2023Will ChatGPT get you caught? Rethinking of Plagiarism Detectioncited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 85; 2024: 70; 2025: 38; 2026: 15.

208 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  4. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  6. 2023Tool Learning with Foundation Modelscited by 302
  7. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  8. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  9. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  10. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  11. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  12. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
38 more
  1. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  5. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  6. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  7. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  8. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  9. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  10. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  11. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  12. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  13. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  14. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  15. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  16. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  17. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  18. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  19. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  20. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  21. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  22. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  23. 2023Is ChatGPT Equipped with Emotional Dialogue Capabilities?cited by 54
  24. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  25. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53
  26. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  27. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
  28. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  29. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  30. 2023SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Promptingcited by 39
  31. 2023GPT-4 Can't Reasoncited by 39
  32. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  33. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37
  34. 2024On the Worst Prompt Performance of Large Language Modelscited by 36
  35. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35
  36. 2023Evaluating Open-QA Evaluationcited by 35
  37. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
  38. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32

50 most cited of 208 All 208 (JSON)