mlsanityData to 7 Oct 2026

Unintended Impacts of LLM Alignment on Global Representation

arXiv 2402.15018 ↗First posted 22 Feb 2024cs.CL cs.CY cs.LGcited by 66

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Before being deployed for user-facing applications, developers align Large Language Models (LLMs) to user preferences through a variety of procedures, such as Reinforcement Learning From Human Feedback (RLHF) and Direct Preference Optimization (DPO). Current evaluations of these procedures focus on benchmarks of instruction following, reasoning, and truthfulness. However, human preferences are not universal, and aligning to specific preference sets may have unintended effects. We explore how alignment impacts performance along three axes of global representation: English dialects, multilingualism, and opinions from and about countries worldwide. Our results show that current alignment procedures create disparities between English dialects and global opinions. We find alignment improves capabilities in several languages. We conclude by discussing design decisions that led to these unintended impacts and recommendations for more equitable preference tuning. We make our code and data publicly available on Github.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 80 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

51 of 80 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  2. 2023Mitigating the Alignment Tax of RLHF12 Sep
  3. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators8 Jul
  4. 2023LIMA: Less Is More for Alignment18 May
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 45

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
33 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2018Model Cards for Model Reportingcited by 1,068
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2023Whose Opinions Do Language Models Reflect?cited by 759
  13. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  14. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  15. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  16. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  17. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  18. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  19. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  20. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  21. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  22. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  23. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  24. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  25. 2023Human Feedback is not Gold Standardcited by 82
  26. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  27. 2023The Foundation Model Transparency Indexcited by 63
  28. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 51
  29. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  30. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  31. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedbackcited by 15
  32. 2023MD3: The Multi-Dialect Dataset of Dialoguescited by 12
  33. 2023Multilingual large language models leak human stereotypes across language boundariescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 26; 2026: 16.

66 papers.

Sort:

By area:

  1. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  2. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  3. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  4. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  5. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 55
  6. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  7. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 47
  8. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  9. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  10. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  11. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  12. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
38 more
  1. 2025The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Modelscited by 28
  2. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
  3. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  4. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  5. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  6. 2025Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Controlcited by 14
  7. 2024Evaluating Dialect Robustness of Language Models via Conversation Understandingcited by 13
  8. 2024The Call for Socially Aware Language Technologiescited by 12
  9. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluationcited by 11
  10. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  11. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
  12. 2024Attributing Culture-Conditioned Generations to Pretraining Corporacited by 8
  13. 2025Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?cited by 8
  14. 2025Benchmarking Overton Pluralism in LLMscited by 7
  15. 2024Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedbackcited by 6
  16. 2025Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages -- A Singlish Case Studycited by 6
  17. 2025LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?cited by 6
  18. 2025Using LLMs to Advance the Cognitive Science of Collectivescited by 6
  19. 2024KGValidator: A Framework for Automatic Validation of Knowledge Graph Constructioncited by 5
  20. 2024Native Design Bias: Studying the Impact of English Nativeness on Language Model Performancecited by 5
  21. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  22. 2025CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimizationcited by 5
  23. 2025DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Contextcited by 5
  24. 2025Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarkscited by 5
  25. 2024Adaptable Moral Stances of Large Language Models on Sexist Content: Implications for Society and Gender Discoursecited by 4
  26. 2025Mind the Gesture: Evaluating AI Sensitivity to Culturally Offensive Non-Verbal Gesturescited by 4
  27. 2025Value Alignment of Social Media Ranking Algorithmscited by 4
  28. 2024"A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMscited by 3
  29. 2025Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Modelscited by 3
  30. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
  31. 2025The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMscited by 2
  32. 2025Culture Cartography: Mapping the Landscape of Cultural Knowledgecited by 2
  33. 2026Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-makingcited by 2
  34. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  35. 2025EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Normscited by 1
  36. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1
  37. 2025Language steering in latent space to mitigate unintended code-switchingcited by 1
  38. 2025Toward Human-Centered AI-Assisted Terminology Workcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On Diversified Preferences of Large Language Model Alignment
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  3. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  6. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  7. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  8. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques