mlsanityData to 7 Oct 2026

Fairness Aware Reward Optimization

arXiv 2602.07799 ↗First posted 8 Feb 2026cs.LG cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Demographic skews in human preference data propagate systematic unfairness through reward models into aligned LLMs. We introduce Fairness Aware Reward Optimization (Faro), an in-processing framework that trains reward models under demographic parity, equalized odds, or counterfactual fairness constraints. We provide the first theoretical analysis of reward-level fairness in LLM alignment, establishing: (i) provable fairness certificates for Faro-trained rewards with controllable slack; a (ii) formal characterization of the accuracy-fairness trade-off induced by KL-regularized fine-tuning, proving fairness transfers from reward to policy; and the (iii) existence of a non-empty Pareto frontier. Unlike pre- and post-processing methods, Faro ensures reward models are simultaneously ordinal (ranking correctly), cardinal (calibrated), and fair. Across multiple LLMs and benchmarks, Faro significantly reduces bias and harmful generations while maintaining or improving model quality.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 77 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

56 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives7 Nov
  2. 2024Self-Rewarding Language Models18 Jan
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
38 more
  1. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  2. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  3. 2014Certifying and removing disparate impactcited by 715
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  6. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  8. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
  9. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  11. 2017On Fairness and Calibrationcited by 406
  12. 2017Algorithmic decision making and the cost of fairnesscited by 402
  13. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  14. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  15. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  16. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 239
  17. 2015The Variational Fair Autoencodercited by 223
  18. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  19. 2019Flexibly Fair Representation Learning by Disentanglementcited by 196
  20. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  21. 2018FairGAN: Fairness-aware Generative Adversarial Networkscited by 164
  22. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  23. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  24. 2018Classification with Fairness Constraints: A Meta-Algorithm with Provable Guaranteescited by 146
  25. 2019On the Fairness of Disentangled Representationscited by 144
  26. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  27. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  28. 2018Differentially Private Fair Learningcited by 76
  29. 2016A Confidence-Based Approach for Balancing Fairness and Accuracycited by 73
  30. 2019Leveraging Labeled and Unlabeled Data for Consistent Fair Binary Classificationcited by 60
  31. 2018Two-Player Games for Efficient Non-Convex Constrained Optimizationcited by 55
  32. 2018Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraintscited by 53
  33. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  34. 2016A statistical framework for fair predictive algorithmscited by 37
  35. 2020Correcting for Selection Bias in Learning-to-rank Systemscited by 35
  36. 2022Bayes-Optimal Classifiers under Group Fairnesscited by 24
  37. 2021Popularity Bias Is Not Always Evil: Disentangling Benign and Harmful Bias for Recommendationcited by 23
  38. 2021Group-Aware Threshold Adaptation for Fair Classificationcited by 21

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
  2. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization
  3. 2025Detecting Prefix Bias in LLM-based Reward Models
  4. 2025Guiding LLM Decision-Making with Fairness Reward Models
  5. 2025Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values
  6. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
  7. 2025Direct Alignment with Heterogeneous Preferences
  8. 2026Procedural Fairness Failures in RLHF from Preference Averaging