mlsanityData to 14 Oct 2026

FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models

arXiv 2503.19540 ↗First posted 25 Mar 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have significantly enhanced interactions between users and models. These advancements concurrently underscore the need for rigorous safety evaluations due to the manifestation of social biases, which can lead to harmful societal impacts. Despite these concerns, existing benchmarks may overlook the intrinsic weaknesses of LLMs, which can generate biased responses even with simple adversarial instructions. To address this critical gap, we introduce a new benchmark, Fairness Benchmark in LLM under Extreme Scenarios (FLEX), designed to test whether LLMs can sustain fairness even when exposed to prompts constructed to induce bias. To thoroughly evaluate the robustness of LLMs, we integrate prompts that amplify potential biases into the fairness assessment. Comparative experiments between FLEX and existing benchmarks demonstrate that traditional evaluations may underestimate the inherent risks in models. This highlights the need for more stringent LLM evaluation benchmarks to guarantee safety and fairness.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

35 of 49 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2023LIMA: Less Is More for Alignmentcited by 1,100
  9. 2021Ethical and social risks of harm from Language Modelscited by 966
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  12. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
23 more
  1. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  4. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  6. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  7. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  8. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  9. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  10. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  11. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  12. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  13. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  14. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  15. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  16. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 116
  17. 2016Crafting Adversarial Input Sequences for Recurrent Neural Networkscited by 103
  18. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  19. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  20. 2023Social Bias Probing: Fairness Benchmarking for Language Modelscited by 32
  21. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  22. 2023Ethical ChatGPT: Concerns, Challenges, and Commandmentscited by 14
  23. 2023DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Biascited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  2. 2025Robustly Improving LLM Fairness in Realistic Settings via Interpretabilitycited by 18
  3. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  4. 2026InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluationcited by 3
  5. 2026TrustLDM: Benchmarking Trustworthiness in Language Diffusion Modelscited by 2
  6. 2026When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Makingcited by 1
  7. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0
  8. 2026Social Bias in Chinese Large Language Models under Social Media Expressionscited by 0