mlsanityData to 7 Oct 2026

FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicity

arXiv 2311.18580 ↗First posted 30 Nov 2023cs.CL cs.CRcited by 22

Not on the map. Why?

Abstract (arXiv)

The widespread of generative artificial intelligence has heightened concerns about the potential harms posed by AI-generated texts, primarily stemming from factoid, unfair, and toxic content. Previous researchers have invested much effort in assessing the harmlessness of generative language models. However, existing benchmarks are struggling in the era of large language models (LLMs), due to the stronger language generation and instruction following capabilities, as well as wider applications. In this paper, we propose FFT, a new benchmark with 2116 elaborated-designed instances, for LLM harmlessness evaluation with factuality, fairness, and toxicity. To investigate the potential harms of LLMs, we evaluate 9 representative LLMs covering various parameter scales, training stages, and creators. Experiments show that the harmlessness of LLMs is still under-satisfactory, and extensive analysis derives some insightful findings that could inspire future research for harmless LLM research.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

42 of 52 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
30 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  6. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  7. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  8. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  9. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  10. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  11. 2023Do Large Language Models Know What They Don't Know?cited by 238
  12. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  13. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  14. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  15. 2021A survey on datasets for fairness-aware machine learningcited by 157
  16. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  17. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  18. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  19. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  20. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  21. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
  22. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoningcited by 49
  23. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 49
  24. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  25. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28
  26. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 23
  27. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  28. 2023Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbotscited by 14
  29. 2023Leveraging Large Language Models for Topic Classification in the Domain of Public Affairscited by 13
  30. 2023Monotonicity for AI ethics and society: An empirical study of the monotonic neural additive model in criminology, education, health care, and financecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 8; 2026: 1.

22 papers.

Sort:

By area:

  1. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  2. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  3. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  5. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  6. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  7. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  8. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  9. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 17
  10. 2024Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Textcited by 14
  11. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  12. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 10
10 more
  1. 2024Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like Onecited by 9
  2. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  3. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  4. 2025ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMscited by 5
  5. 2024Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoningcited by 4
  6. 2026Optimal Transport for LLM Reward Modeling from Noisy Preferencecited by 3
  7. 2025FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCscited by 1
  8. 2024Fine-Grained Detoxification via Instance-Level Prefixes for Large Language Modelscited by 0
  9. 2025Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectivescited by 0
  10. 2025InvThink: Premortem Reasoning for Safer Language Modelscited by 0