mlsanityData to 7 Oct 2026

LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases

arXiv 2501.03112 ↗First posted 6 Jan 2025cs.CL cs.AI cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have been observed to exhibit bias in numerous ways, potentially creating or worsening outcomes for specific groups identified by protected attributes such as sex, race, sexual orientation, or age. To help address this gap, we introduce LangFair, an open-source Python package that aims to equip LLM practitioners with the tools to evaluate bias and fairness risks relevant to their specific use cases. The package offers functionality to easily generate evaluation datasets, comprised of LLM responses to use-case-specific prompts, and subsequently calculate applicable metrics for the practitioner's use case. To guide in metric selection, LangFair offers an actionable decision framework.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

31 of 40 references.

References · 31

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  4. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  5. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  6. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  9. 2018Gender Bias in Coreference Resolutioncited by 438
  10. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  11. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  12. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
19 more
  1. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  2. 2019The What-If Tool: Interactive Probing of Machine Learning Modelscited by 207
  3. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  4. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  5. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  6. 2018Aequitas: A Bias and Fairness Audit Toolkitcited by 145
  7. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  8. 2022Holistic Evaluation of Language Modelscited by 115
  9. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 113
  10. 2020Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Biascited by 82
  11. 2023Fairlearn: Assessing and Improving Fairness of AI Systemscited by 74
  12. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  13. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  14. 2022Perturbation Augmentation for Fairer NLPcited by 51
  15. 2023WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Modelscited by 42
  16. 2020LiFT: A Scalable Framework for Measuring Fairness in ML Applicationscited by 22
  17. 2022Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Resultscited by 17
  18. 2023Towards Auditing Large Language Models: Improving Text-based Stereotype Detectioncited by 12
  19. 2024Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025GAAPO: Genetic Algorithmic Applied to Prompt Optimizationcited by 10
  2. 2025Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directionscited by 3
  3. 2025Uncovering Representation Bias for Investment Decisions in Open-Source Large Language Modelscited by 2
  4. 2025Fairness Evaluation and Inference Level Mitigation in LLMscited by 2
  5. 2026FairLMs: A Turnkey Library for Fairness in Language Modelscited by 0