mlsanityData to 14 Oct 2026

TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models

arXiv 2306.11507 ↗First posted 20 Jun 2023cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) such as ChatGPT, have gained significant attention due to their impressive natural language processing capabilities. It is crucial to prioritize human-centered principles when utilizing these models. Safeguarding the ethical and moral compliance of LLMs is of utmost importance. However, individual ethical issues have not been well studied on the latest LLMs. Therefore, this study aims to address these gaps by introducing a new benchmark -- TrustGPT. TrustGPT provides a comprehensive evaluation of LLMs in three crucial areas: toxicity, bias, and value-alignment. Initially, TrustGPT examines toxicity in language models by employing toxic prompt templates derived from social norms. It then quantifies the extent of bias in models by measuring quantifiable toxicity values across different groups. Lastly, TrustGPT assesses the value of conversation generation models from both active value-alignment and passive value-alignment tasks. Through the implementation of TrustGPT, this research aims to enhance our understanding of the performance of conversation generation models and promote the development of language models that are more ethical and socially responsible.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

43 of 61 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  12. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
31 more
  1. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  2. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  3. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  5. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  6. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  7. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  8. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  9. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  11. 2019Measuring Bias in Contextualized Word Representationscited by 276
  12. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  13. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  14. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  15. 2018Gender Bias in Neural Natural Language Processingcited by 211
  16. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  17. 2021Challenges in Detoxifying Language Modelscited by 169
  18. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  19. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  20. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
  21. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  22. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  23. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 53
  24. 2023An Empirical Study on Information Extraction using Large Language Modelscited by 46
  25. 2022Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversalcited by 40
  26. 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
  27. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
  28. 2022Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimizationcited by 33
  29. 2021Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Modelscited by 20
  30. 2023Semantic Compression With Large Language Modelscited by 17
  31. 2022Enabling Classifiers to Make Judgements Explicitly Aligned with Human Valuescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 30; 2025: 14; 2026: 5.

58 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  5. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  8. 2023Large Language Model Alignment: A Surveycited by 243
  9. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  10. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  11. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  12. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
38 more
  1. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  2. 2024Large Language Model Safety: A Holistic Surveycited by 59
  3. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  4. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  5. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  6. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  7. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  8. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  9. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  10. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  11. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 18
  12. 2024CEB: Compositional Evaluation Benchmark for Fairness in Large Language Modelscited by 18
  13. 2024Causal-Guided Active Learning for Debiasing Large Language Modelscited by 15
  14. 2025When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmascited by 15
  15. 2024PersoBench: Benchmarking Personalized Response Generation in Large Language Modelscited by 13
  16. 2024Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like Onecited by 11
  17. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 11
  18. 2024BenchmarkCards: Standardized Documentation for Large Language Model Benchmarkscited by 10
  19. 2024Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignmentcited by 10
  20. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 9
  21. 2024GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Mapscited by 8
  22. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  23. 2024Fairness in Large Language Models in Three Hourscited by 7
  24. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Modelscited by 6
  25. 2025Artificial Intelligence in Spectroscopy: Advancing Chemistry from Prediction to Generation and Beyondcited by 6
  26. 2025Digital Player: Evaluating Large Language Models based Human-like Agent in Gamescited by 6
  27. 2024Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approachcited by 5
  28. 2025LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Casescited by 5
  29. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  30. 2024Strong Preferences Affect the Robustness of Preference Models and Value Alignmentcited by 4
  31. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  32. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  33. 2023Risk Aware Benchmarking of Large Language Modelscited by 2
  34. 2024Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarkingcited by 2
  35. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
  36. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  37. 2024Hidden in Plain Sight: Exploring Chat History Tampering in Interactive Language Modelscited by 1
  38. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 1

50 most cited of 58 All 58 (JSON)