mlsanityData to 7 Oct 2026

SafeText: A Benchmark for Exploring Physical Safety in Language Models

arXiv 2210.10045 ↗First posted 18 Oct 2022cs.CL cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

Understanding what constitutes safe text is an important issue in natural language processing and can often prevent the deployment of models deemed harmful and unsafe. One such type of safety that has been scarcely studied is commonsense physical safety, i.e. text that is not explicitly violent and requires additional commonsense knowledge to comprehend that it leads to physical harm. We create the first benchmark dataset, SafeText, comprising real-life scenarios with paired safe and physically unsafe pieces of advice. We utilize SafeText to empirically study commonsense physical safety across various models designed for text generation and commonsense reasoning tasks. We find that state-of-the-art large language models are susceptible to the generation of unsafe text and have difficulty rejecting unsafe advice. As a result, we argue for further studies of safety and the assessment of commonsense physical safety in models before release.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

25 of 43 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2020Aligning AI With Shared Human Valuescited by 706
13 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  2. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  3. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  4. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  5. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  6. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 92
  7. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  8. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  9. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  10. 2022Mitigating Covertly Unsafe Text within Natural Language Systemscited by 13
  11. 2018Detecting Gang-Involved Escalation on Social Media Using Contextcited by 5
  12. 2021SkillBot: Identifying Risky Content for Children in Alexa Skillscited by 3
  13. 2022'Beach' to 'Bitch': Inadvertent Unsafe Transcription of Kids' Content on YouTubecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 10; 2025: 8; 2026: 5.

41 papers.

Sort:

By area:

  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  2. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  4. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  5. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  6. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  7. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  10. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  11. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  12. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
29 more
  1. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  2. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  3. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMscited by 22
  4. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 17
  5. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14
  6. 2023Building Trustworthy NeuroSymbolic AI Systems: Consistency, Reliability, Explainability, and Safetycited by 14
  7. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Modelscited by 12
  8. 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Taskscited by 8
  9. 2025FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Modelscited by 7
  10. 2025IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languagescited by 7
  11. 2023Similarity-weighted Construction of Contextualized Commonsense Knowledge Graphs for Knowledge-intense Argumentation Taskscited by 6
  12. 2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6
  13. 2022Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AIcited by 5
  14. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  15. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  16. 2024SafeLLM: Domain-Specific Safety Monitoring for Large Language Models: A Case Study of Offshore Wind Maintenancecited by 4
  17. 2023Attribute Diversity Determines the Systematicity Gap in VQAcited by 3
  18. 2024CHBench: A Chinese Dataset for Evaluating Health in Large Language Modelscited by 3
  19. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Modelscited by 3
  20. 2024Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capabilitycited by 2
  21. 2024SoK: Towards Security and Safety of Edge AIcited by 1
  22. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  23. 2026AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albaniancited by 1
  24. 2026PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warningcited by 1
  25. 2023Evaluating and Improving Value Judgments in AI: A Scenario-Based Study on Large Language Models' Depiction of Social Conventionscited by 0
  26. 2024XTRUST: On the Multilingual Trustworthiness of Large Language Modelscited by 0
  27. 2025OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Modelscited by 0
  28. 2026When Words Are Safe But Actions Kill: Probing Physical Jailbreak Beyond Textual Jailbreak in Hidden-State Risk Spacecited by 0
  29. 2026From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
  2. 2026SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond
  3. 2026Beyond Content Safety: Real-Time Monitoring for Reasoning Vulnerabilities in Large Language Models
  4. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
  5. 2025MetaSC: Test-Time Safety Specification Optimization for Language Models
  6. 2026When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life
  7. 2025SafeRBench: Dissecting the Reasoning Safety of Large Language Models
  8. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness