mlsanityData to 14 Oct 2026

Guiding LLM Decision-Making with Fairness Reward Models

arXiv 2507.11344 ↗First posted 15 Jul 2025cs.LGcited by 9

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Large language models are increasingly used to support high-stakes decisions, potentially influencing who is granted bail or receives a loan. Naive chain-of-thought sampling can improve average decision accuracy, but has also been shown to amplify unfair bias. To address this challenge and enable the trustworthy use of reasoning models in high-stakes decision-making, we propose a framework for training a generalizable Fairness Reward Model (FRM). Our model assigns a fairness score to LLM reasoning, enabling the system to down-weight biased trajectories and favor equitable ones when aggregating decisions across reasoning chains. We show that a single Fairness Reward Model, trained on weakly supervised, LLM-annotated examples of biased versus unbiased reasoning, transfers across tasks, domains, and model families without additional fine-tuning. Applied to real-world decision-making tasks including recidivism prediction and social media moderation, we show that our approach consistently improves fairness while matching, or even surpassing, baseline accuracy.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 42 of its 60 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

42 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models21 May
  2. 2024Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting28 Jan

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
28 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  3. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  4. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  5. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  6. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  7. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  8. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  9. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  10. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  11. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  12. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  13. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  14. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  15. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  16. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  17. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
  18. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  19. 2024Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processescited by 39
  20. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  21. 2023Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implicationscited by 29
  22. 2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14
  23. 2024Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Modelscited by 12
  24. 2025DecisionFlow: Advancing Large Language Model as Principled Decision Makercited by 12
  25. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  26. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
  27. 2024Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Modelscited by 2
  28. 2024Exploring Accuracy-Fairness Trade-off in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  2. 2025'Rich Dad, Poor Lad': How do Large Language Models Contextualize Socioeconomic Factors in College Admission ?cited by 1
  3. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMscited by 1
  4. 2026The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMscited by 1
  5. 2026Robust LLM Performance Certification via Constrained Maximum Likelihood Estimationcited by 0
  6. 2026It Takes One to Bias Them All: Breaking Bad with One-Shot GRPOcited by 0
  7. 2026Wait, am I Being Fair? Characterizing Deductive Stereotyping and Mitigating It with Fair-GCGcited by 0
  8. 2026One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMscited by 0
  9. 2026Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models
  2. 2025FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks
  3. 2026Fair Like Us? Auditing LLM Alignment in Resource Allocation
  4. 2025Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning
  5. 2024Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification
  6. 2025Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models
  7. 2026Fairness Aware Reward Optimization
  8. 2021Fairness-aware Summarization for Justified Decision-Making