mlsanityData to 7 Oct 2026

Sociotechnical Safety Evaluation of Generative AI Systems

arXiv 2310.11986 ↗First posted 18 Oct 2023cs.AI cs.CL cs.CYcited by 154

Not on the map. Why?

Abstract (arXiv)

Generative AI systems produce a range of risks. To ensure the safety of generative AI systems, these risks must be evaluated. In this paper, we make two main contributions toward establishing such evaluations. First, we propose a three-layered framework that takes a structured, sociotechnical approach to evaluating these risks. This framework encompasses capability evaluations, which are the main current approach to safety evaluation. It then reaches further by building on system safety principles, particularly the insight that context determines whether a given capability may cause harm. To account for relevant context, our framework adds human interaction and systemic impacts as additional layers of evaluation. Second, we survey the current state of safety evaluation of generative AI systems and create a repository of existing evaluations. Three salient evaluation gaps emerge from this analysis. We propose ways forward to closing these gaps, outlining practical steps as well as roles and responsibilities for different actors. Sociotechnical safety evaluation is a tractable approach to the robust and comprehensive safety evaluation of generative AI systems.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). Builds on 148 of its 398 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

148 of 398 references.

References · 148

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  14. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  15. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  16. 2023PaLM 2 Technical Reportcited by 1,154
  17. 2022Red Teaming Language Models with Language Modelscited by 983
  18. 2021Ethical and social risks of harm from Language Modelscited by 957
  19. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  20. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  21. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  22. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  23. 2023A Watermark for Large Language Modelscited by 667
  24. 2018Object Hallucination in Image Captioningcited by 661
  25. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  26. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  27. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  28. 2020Jukebox: A Generative Model for Musiccited by 535
  29. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  30. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  31. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  32. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  33. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  34. 2023MusicLM: Generating Music From Textcited by 442
  35. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  36. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  37. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  38. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340

50 most cited of 148 All 148 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 64; 2025: 48; 2026: 40.

154 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  3. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  4. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  5. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  6. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  7. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  8. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  9. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  10. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  11. 2024Visibility into AI Agentscited by 77
  12. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
38 more
  1. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66
  2. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  3. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  4. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  5. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  6. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  7. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  8. 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
  9. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  10. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  11. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  12. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 32
  13. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  14. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  15. 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 27
  16. 2025Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Modelscited by 25
  17. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  18. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  19. 2024Can Language Models Recognize Convincing Arguments?cited by 21
  20. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  21. 2025Human Decision-making is Susceptible to AI-driven Manipulationcited by 19
  22. 2024Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generatorscited by 18
  23. 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
  24. 2024IDs for AI Systemscited by 18
  25. 2024The 2024 Foundation Model Transparency Indexcited by 18
  26. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
  27. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
  28. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 17
  29. 2024Can LLMs make trade-offs involving stipulated pain and pleasure states?cited by 16
  30. 2024Generative Ghosts: Anticipating Benefits and Risks of AI Afterlivescited by 15
  31. 2024Harm Amplification in Text-to-Image Modelscited by 14
  32. 2024Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Researchcited by 14
  33. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  34. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
  35. 2024Large language models can consistently generate high-quality content for election disinformation operationscited by 13
  36. 2024The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligencecited by 13
  37. 2024Acceptable Use Policies for Foundation Modelscited by 12
  38. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12

50 most cited of 154 All 154 (JSON)