mlsanityData to 14 Oct 2026

Fair on the Surface? Benchmarking Hidden-Output Fairness Gaps in LLM Recommenders

arXiv 2608.08284 ↗First posted 8 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Fairness audits for LLM-based recommenders have largely focused on observable outputs, implicitly assuming that stable recommendations reflect stable internal processing. We challenge this assumption with FairGap, the first benchmark to jointly evaluate recommendation fairness at two levels: observable output shift (OBS) and hidden representation shift (IBS), measured through controlled counterfactual identity probes across gender, age, and race. Their relationship is summarized via Representation-Output Alignment (ROA), with quadrant diagnostics for identifying user-level hidden-output mismatch. Applied to six open-weight LLM families across three domains, FairGap reveals pervasive hidden-output decoupling: ROA rarely exceeds 0.22, and a non-negligible user population shows stable outputs despite substantial internal shifts, a mode that output-only audits cannot detect by design. Further, activation steering that reduces IBS by up to 8x simultaneously worsens OBS, demonstrating a fundamental tension between internal and output-level fairness that existing frameworks are unequipped to diagnose.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

34 of 43 references.

References · 34

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2021Ethical and social risks of harm from Language Modelscited by 966
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  9. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  11. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  12. 2023Large Language Models for Information Retrieval: A Surveycited by 331
22 more
  1. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  2. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  3. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 114
  4. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  5. 2023A Survey on Fairness in Large Language Modelscited by 95
  6. 2023Large Language Models for Generative Recommendation: A Survey and Visionary Discussionscited by 73
  7. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
  8. 2023UP5: Unbiased Foundation Model for Fairness-aware Recommendationcited by 38
  9. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  10. 2023Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projectioncited by 25
  11. 2023Fairness and Diversity in Recommender Systems: A Surveycited by 22
  12. 2025System Prompt Optimization with Meta-Learningcited by 18
  13. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
  14. 2024Fairness Definitions in Language Models Explainedcited by 11
  15. 2024Fairness in Large Language Models in Three Hourscited by 7
  16. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  17. 2024User-item fairness tradeoffs in recommendationscited by 5
  18. 2025Stairway to Fairness: Connecting Group and Individual Fairnesscited by 3
  19. 2026Auditing Language Model Unlearning via Information Decompositioncited by 3
  20. 2025Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMscited by 2
  21. 2025Envy-Free but Still Unfair: Envy-Freeness Up To One Item (EF-1) in Personalized Recommendationcited by 1
  22. 2026Measuring Individual User Fairness with User Similarity and Effectiveness Disparitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Fairness in LLM-Based Recommender Systems: A Survey
  2. 2026Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems
  3. 2026Fairness Begins with State: Purifying Latent Preferences for Hierarchical Reinforcement Learning in Interactive Recommendation
  4. 2025FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems
  5. 2026Lightweight Fairness for LLM-Based Recommendations via Kernelized Projection and Gated Adapters
  6. 2025UFO: Unfair-to-Fair Evolving Mitigates Unfairness in LLM-based Recommender Systems via Self-Play Fine-tuning ↗
  7. 2025Improving Recommendation Fairness without Sensitive Attributes Using Multi-Persona LLMs
  8. 2022Causal Intervention for Fairness in Multi-behavior Recommendation