mlsanityData to 7 Oct 2026

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models

arXiv 2306.11698 ↗First posted 20 Jun 2023cs.CL cs.AI cs.CRcited by 505

Not on the map. Why?

Cited by 114 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives -- including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially because GPT-4 follows (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/ ; our dataset can be previewed at https://huggingface.co/datasets/AI-Secure/DecodingTrust ; a concise version of this work is at https://openreview.net/pdf?id=kaHpo8OZw2 .

Key facts. Cited by 505 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 238; 2025: 128; 2026: 102.

505 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  6. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  12. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
38 more
  1. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  2. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  4. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  5. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  6. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  7. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  8. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  9. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  10. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  11. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  12. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  13. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  14. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  15. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  16. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  17. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  18. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  19. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  20. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  21. 2023A Survey on Fairness in Large Language Modelscited by 91
  22. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  23. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  24. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  25. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  26. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  27. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  28. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  29. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  30. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  31. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  32. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  33. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  34. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  35. 2024Safeguarding Large Language Models: A Surveycited by 73
  36. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  37. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  38. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70

50 most cited of 505 All 505 (JSON)