mlsanityData to 14 Oct 2026

In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT

arXiv 2304.08979 ↗First posted 18 Apr 2023cs.CR cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

The way users acquire information is undergoing a paradigm shift with the advent of ChatGPT. Unlike conventional search engines, ChatGPT retrieves knowledge from the model itself and generates answers for users. ChatGPT's impressive question-answering (QA) capability has attracted more than 100 million users within a short period of time but has also raised concerns regarding its reliability. In this paper, we perform the first large-scale measurement of ChatGPT's reliability in the generic QA scenario with a carefully curated set of 5,695 questions across ten datasets and eight domains. We find that ChatGPT's reliability varies across different domains, especially underperforming in law and science questions. We also demonstrate that system roles, originally designed by OpenAI to allow users to steer ChatGPT's behavior, can impact ChatGPT's reliability in an imperceptible way. We further show that ChatGPT is vulnerable to adversarial examples, and even a single character change can negatively affect its reliability in certain cases. We believe that our study provides valuable insights into ChatGPT's reliability and underscores the need for strengthening the reliability and security of large language models (LLMs).

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 50 of its 78 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

50 of 78 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
38 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  7. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  8. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  10. 2019ELI5: Long Form Question Answeringcited by 575
  11. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  12. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  13. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  14. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  15. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  16. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  17. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  18. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  19. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  20. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
  21. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  22. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  23. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  24. 2023A Categorical Archive of ChatGPT Failurescited by 168
  25. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  26. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 110
  27. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 101
  28. 2023MGTBench: Benchmarking Machine-Generated Text Detectioncited by 93
  29. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  30. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  31. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  32. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  33. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 73
  34. 2023Consistency Analysis of ChatGPTcited by 37
  35. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 23
  36. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  37. 2018Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehensioncited by 19
  38. 2023Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnesscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 9; 2025: 6; 2026: 3.

37 papers.

Sort:

By area:

  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  2. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  3. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  4. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  5. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  6. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  7. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  8. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  9. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 82
  10. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 63
  11. 2024Instruction Backdoor Attacks Against Customized LLMscited by 62
  12. 2023Do Large Language Models Know about Facts?cited by 51
25 more
  1. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 48
  2. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 42
  3. 2024Prompt Stealing Attacks Against Large Language Modelscited by 40
  4. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  5. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 31
  6. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  7. 2023Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptionscited by 20
  8. 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
  9. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 20
  10. 2024Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perceptioncited by 17
  11. 2025Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrencycited by 17
  12. 2025Language Model Uncertainty Quantification with Attention Chaincited by 14
  13. 2023AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topicscited by 12
  14. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  15. 2023Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Contentcited by 9
  16. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 8
  17. 2023How many words does ChatGPT know? The answer is ChatWordscited by 6
  18. 2024Zero-Shot Topic Classification of Column Headers: Leveraging LLMs for Metadata Enrichmentcited by 5
  19. 2025Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applicationscited by 4
  20. 2026Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarkscited by 4
  21. 2026Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Taskscited by 3
  22. 2023Comprehensive Assessment of Toxicity in ChatGPTcited by 2
  23. 2025BitHydra: Towards Bit-flip Inference Cost Attack against Large Language Modelscited by 2
  24. 2023Applying Standards to Advance Upstream & Downstream Ethics in Large Language Modelscited by 0
  25. 2026Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Modelscited by 0