Human Preferences for Constructive Interactions in Language Model Alignment
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
As large language models (LLMs) enter the mainstream, aligning them to foster constructive dialogue rather than exacerbate societal divisions is critical. Using an individualized and multicultural alignment dataset of over 7,500 conversations of individuals from 74 countries engaging with 21 LLMs, we examined how linguistic attributes linked to constructive interactions are reflected in human preference data used for training AI. We found that users consistently preferred well-reasoned and nuanced responses while rejecting those high in personal storytelling. However, users who believed that AI should reflect their values tended to place less preference on reasoning in LLM responses and more on curiosity. Encouragingly, we observed that users could set the tone for how constructive their conversation would be, as LLMs mirrored linguistic attributes, including toxicity, in user queries.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 26 references that are in the map. First posted on arXiv 5 Mar 2025.
Builds on
16 of 26 references.
Closest ancestors · 3
Same direction, newest first
- 2025C3AI: Crafting and Evaluating Constitutions for Constitutional AI21 Feb
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
- 2024Dissecting Human and LLM Preferences17 Feb
Other references · 13
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
- 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
- 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
- 2023Generative Language Models Exhibit Social Identity Biasescited by 111
- 2021Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversationscited by 20
- 2023Bridging Systems: Open Problems for Countering Destructive Divisiveness across Ranking, Recommenders, and Governancecited by 17
- 2023Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safetycited by 8
- 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Aligning LLMs with Individual Preferences via Interaction
- 2024A Survey on Human Preference Learning for Large Language Models
- 2024Aligning language models with human preferences
- 2026Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models
- 2024Uncovering Factor Level Preferences to Improve Human-Model Alignment
- 2025Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking
- 2024Understanding the Learning Dynamics of Alignment with Human Feedback
- 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior