# SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use

arXiv: https://arxiv.org/abs/2505.17332 · HTML: https://mlsanity.com/paper/2505.17332/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2025.

## Built on by (most cited first, 15 of 15)

- [Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems](https://mlsanity.com/paper/2505.18366/) (2025)
- [Aligning LLMs for Multilingual Consistency in Enterprise Applications](https://mlsanity.com/paper/2509.23659/) (2025)
- [Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs](https://mlsanity.com/paper/2510.23854/) (2025)
- [Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMs](https://mlsanity.com/paper/2508.15831/) (2025)
- [SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models](https://mlsanity.com/paper/2509.14270/) (2025)
- [AccessEval: Benchmarking Disability Bias in Large Language Models](https://mlsanity.com/paper/2509.22703/) (2025)
- [IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages](https://mlsanity.com/paper/2602.16832/) (2026)
- [IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia](https://mlsanity.com/paper/2603.17915/) (2026)
- [LLM-Guided Lifecycle-Aware Clustering of Multi-Turn Customer Support Conversations](https://mlsanity.com/paper/2601.04388/) (2026)
- [Anthropogenic Regional Adaptation in Multimodal Vision-Language Model](https://mlsanity.com/paper/2604.11490/) (2026)
- [PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications](https://mlsanity.com/paper/2509.23879/) (2025)
- [BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture](https://mlsanity.com/paper/2511.03180/) (2025)
- [Who Gets Left Behind? Auditing Disability Inclusivity in Large Language Models](https://mlsanity.com/paper/2509.00963/) (2025)
- [JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data](https://mlsanity.com/paper/2605.24414/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)

## Abstract (arXiv)

Enterprise customers are increasingly adopting Large Language Models (LLMs) for critical communication tasks, such as drafting emails, crafting sales pitches, and composing casual messages. Deploying such models across different regions requires them to understand diverse cultural and linguistic contexts and generate safe and respectful responses. For enterprise applications, it is crucial to mitigate reputational risks, maintain trust, and ensure compliance by effectively identifying and handling unsafe or offensive language. To address this, we introduce SweEval, a benchmark simulating real-world scenarios with variations in tone (positive or negative) and context (formal or informal). The prompts explicitly instruct the model to include specific swear words while completing the task. This benchmark evaluates whether LLMs comply with or resist such inappropriate instructions and assesses their alignment with ethical frameworks, cultural nuances, and language comprehension capabilities. In order to advance research in building ethically aligned AI systems for enterprise use and beyond, we release the dataset and code: https://github.com/amitbcp/multilingual_profanity.