# Unintended Impacts of LLM Alignment on Global Representation

arXiv: https://arxiv.org/abs/2402.15018 · HTML: https://mlsanity.com/paper/2402.15018/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 51 of its 80 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 51)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)

## Built on by (most cited first, 30 of 66)

- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination](https://mlsanity.com/paper/2406.08818/) (2024)
- [CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting](https://mlsanity.com/paper/2404.10199/) (2024)
- [Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees](https://mlsanity.com/paper/2405.10301/) (2024)
- [NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models](https://mlsanity.com/paper/2404.12464/) (2024)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [Aligning Language Models with Demonstrated Feedback](https://mlsanity.com/paper/2406.00888/) (2024)
- [From Distributional to Overton Pluralism: Investigating Large Language Model Alignment](https://mlsanity.com/paper/2406.17692/) (2024)
- [Preference Tuning For Toxicity Mitigation Generalizes Across Languages](https://mlsanity.com/paper/2406.16235/) (2024)
- [Self-Pluralising Culture Alignment for Large Language Models](https://mlsanity.com/paper/2410.12971/) (2024)
- [The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models](https://mlsanity.com/paper/2502.11266/) (2025)
- [An Evaluation of Cultural Value Alignment in LLM](https://mlsanity.com/paper/2504.08863/) (2025)
- [CulturePark: Boosting Cross-cultural Understanding in Large Language Models](https://mlsanity.com/paper/2405.15145/) (2024)
- [Rejected Dialects: Biases Against African American Language in Reward Models](https://mlsanity.com/paper/2502.12858/) (2025)
- [CARE: Multilingual Human Preference Learning for Cultural Awareness](https://mlsanity.com/paper/2504.05154/) (2025)
- [Evaluating Dialect Robustness of Language Models via Conversation Understanding](https://mlsanity.com/paper/2405.05688/) (2024)
- [Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Control](https://mlsanity.com/paper/2504.17130/) (2025)
- [The Call for Socially Aware Language Technologies](https://mlsanity.com/paper/2405.02411/) (2024)
- [WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation](https://mlsanity.com/paper/2410.12722/) (2024)
- [Value Drifts: Tracing Value Alignment During LLM Post-Training](https://mlsanity.com/paper/2510.26707/) (2025)
- [Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models](https://mlsanity.com/paper/2406.16135/) (2024)
- [Attributing Culture-Conditioned Generations to Pretraining Corpora](https://mlsanity.com/paper/2412.20760/) (2024)
- [Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?](https://mlsanity.com/paper/2507.13490/) (2025)
- [Benchmarking Overton Pluralism in LLMs](https://mlsanity.com/paper/2512.01351/) (2025)
- [Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback](https://mlsanity.com/paper/2406.05902/) (2024)
- [Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages -- A Singlish Case Study](https://mlsanity.com/paper/2502.12485/) (2025)
- [LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?](https://mlsanity.com/paper/2503.15003/) (2025)
- [CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimization](https://mlsanity.com/paper/2504.08820/) (2025)

## Abstract (arXiv)

Before being deployed for user-facing applications, developers align Large Language Models (LLMs) to user preferences through a variety of procedures, such as Reinforcement Learning From Human Feedback (RLHF) and Direct Preference Optimization (DPO). Current evaluations of these procedures focus on benchmarks of instruction following, reasoning, and truthfulness. However, human preferences are not universal, and aligning to specific preference sets may have unintended effects. We explore how alignment impacts performance along three axes of global representation: English dialects, multilingualism, and opinions from and about countries worldwide. Our results show that current alignment procedures create disparities between English dialects and global opinions. We find alignment improves capabilities in several languages. We conclude by discussing design decisions that led to these unintended impacts and recommendations for more equitable preference tuning. We make our code and data publicly available on Github.