mlsanityData to 7 Oct 2026

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration

arXiv 2406.15951 ↗First posted 22 Jun 2024cs.CLcited by 111

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

While existing alignment paradigms have been integral in developing large language models (LLMs), LLMs often learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities. We propose Modular Pluralism, a modular framework based on multi-LLM collaboration for pluralistic alignment: it "plugs into" a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional. Modular Pluralism is uniquely compatible with black-box LLMs and offers the modular control of adding new community LMs for previously underrepresented communities. We evaluate Modular Pluralism with six tasks and four datasets featuring questions/instructions with value-laden and perspective-informed responses. Extensive experiments demonstrate that Modular Pluralism advances the three pluralism objectives across six black-box and open-source LLMs. Further analysis reveals that LLMs are generally faithful to the inputs from smaller community LLMs, allowing seamless patching by adding a new community LM to better cover previously underrepresented communities.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 80 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

26 of 80 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
  2. 2024IterAlign: Iterative Constitutional Alignment of Large Language Models27 Mar
  3. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation8 Feb
  4. 2024A Roadmap to Pluralistic Alignment7 Feb
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  6. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Mistral 7Bcited by 3,065
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  11. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  12. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
8 more
  1. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  2. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  4. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  5. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  6. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 49
  7. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 47
  8. 2023Aligning Large Language Models with Human: A Surveycited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 51; 2026: 44.

111 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  3. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  4. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
  5. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  6. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  7. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
  8. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  9. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 26
  10. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  11. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  12. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
38 more
  1. 2025AI Alignment at Your Discretioncited by 20
  2. 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 20
  3. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  4. 2025VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcarecited by 17
  5. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
  6. 2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
  7. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  8. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  9. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  10. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  11. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  12. 2025CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectivescited by 10
  13. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  14. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  15. 2025Don't Throw Away Your Pretrained Modelcited by 9
  16. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  17. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  18. 2025Many LLMs Are More Utilitarian Than Onecited by 7
  19. 2025The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Modelscited by 7
  20. 2025Just-In-Time Objectives: A General Approach for Specialized AI Interactionscited by 7
  21. 2025Benchmarking Overton Pluralism in LLMscited by 7
  22. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  23. 2024SPICA: Retrieving Scenarios for Pluralistic In-Context Alignmentcited by 6
  24. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  25. 2025Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regressioncited by 6
  26. 2026Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responsescited by 6
  27. 2024Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviorscited by 5
  28. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  29. 2025Sparsity May Be All You Need: Sparse Random Parameter Adaptationcited by 5
  30. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  31. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  32. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  33. 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
  34. 2024Understanding The Effect Of Temperature On Alignment With Human Opinionscited by 4
  35. 2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4
  36. 2025Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Modelscited by 4
  37. 2025Value Alignment of Social Media Ranking Algorithmscited by 4
  38. 2024Different Bias Under Different Criteria: Assessing Bias in LLMs with a Fact-Based Approachcited by 3

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset
  2. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
  3. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
  4. 2024Aligning LLMs with Individual Preferences via Interaction
  5. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  6. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
  7. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs
  8. 2026Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment