mlsanityData to 14 Oct 2026

TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment

arXiv 2606.01755 ↗First posted 1 Jun 2026cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalized large language models adapt responses to users' preferences and social attributes, but can introduce substantial universal truth inconsistencies across social groups, where some groups systematically receive less accurate responses on objective tasks. Existing alignment methods either ignore personalization or mainly focus on subjective preference alignment, largely overlooking fairness and consistency in universal truths. To address this gap, we study Truth-Invariant Alignment (TIA), an alignment problem for personalized LLMs that aims to ensure universal truths remain consistent across social groups while preserving personalization. We propose TriAlign, the first offline multi-agent reinforcement learning (MARL) framework for TIA, where each social group is modeled as an agent interacting. TriAlign jointly optimizes universal truth accuracy, cross-group truth consistency, and personalization through a fairness-aware objective and an explicit inconsistency penalty. Experiments across diverse benchmarks demonstrate that TriAlign achieves a stronger balance among these three objectives than strong baselines, reducing universal truth disparities across social groups while improving both objective task performance and personalization quality.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 55 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

32 of 55 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  9. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  10. 2025LIMO: Less is More for Reasoningcited by 498
  11. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  12. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
20 more
  1. 2024Measuring short-form factuality in large language modelscited by 373
  2. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  3. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  4. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  5. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  6. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 199
  7. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  8. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  9. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  10. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  11. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  12. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  13. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  14. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 57
  15. 2024Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Modelscited by 45
  16. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 27
  17. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 18
  18. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  19. 2021Accurate, yet inconsistent? Consistency Analysis on Language Understanding Modelscited by 13
  20. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
  2. 2024Aligning LLMs with Individual Preferences via Interaction
  3. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback
  4. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  5. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  6. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  7. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
  8. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferences