mlsanityData to 7 Oct 2026

Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation

arXiv 2402.05699 ↗First posted 8 Feb 2024cs.CL cs.AI cs.CYcited by 46

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse. Drawing from the sociological insight that acknowledging all parties' concerns is a key factor in shaping human values, this paper proposes a novel direction to align LLMs by themselves: social scene simulation. To achieve this, we present MATRIX, a novel social scene simulator that emulates realistic scenes around a user's input query, enabling the LLM to take social consequences into account before responding. MATRIX serves as a virtual rehearsal space, akin to a Monopolylogue, where the LLM performs diverse roles related to the query and practice by itself. To inject this alignment, we fine-tune the LLM with MATRIX-simulated data, ensuring adherence to human values without compromising inference speed. We theoretically show that the LLM with MATRIX outperforms Constitutional AI under mild assumptions. Finally, extensive experiments validate that our method outperforms over 10 baselines across 4 benchmarks. As evidenced by 875 user ratings, our tuned 13B-size LLM exceeds GPT-4 in aligning with human values. See our project page at https://shuotang123.github.io/MATRIX.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 65 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

42 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  2. 2023Training Socially Aligned Language Models on Simulated Social Interactions26 May
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023Mistral 7Bcited by 3,065
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
27 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  13. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  14. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  15. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  16. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  17. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  18. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  19. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  20. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  21. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  22. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  23. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  24. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
  25. 2023Managing extreme AI risks amid rapid progresscited by 48
  26. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  27. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 26; 2026: 3.

46 papers.

Sort:

By area:

  1. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 180
  2. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  3. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  4. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  5. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  6. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Developmentcited by 73
  7. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  8. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  9. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  10. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  11. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  12. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
34 more
  1. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  2. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  3. 2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineeringcited by 26
  4. 2024Synthesizing Post-Training Data for LLMs through Multi-Agent Simulationcited by 22
  5. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  6. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  7. 2024Plurals: A System for Guiding LLMs Via Simulated Social Ensemblescited by 17
  8. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  9. 2025MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systemscited by 14
  10. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  11. 2025SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulationcited by 10
  12. 2024ProgressGym: Alignment with a Millennium of Moral Progresscited by 9
  13. 2025PathVLM-R1: A Reinforcement Learning-Driven Reasoning Model for Pathology Visual-Language Taskscited by 8
  14. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  15. 2025Infusing Theory of Mind into Socially Intelligent LLM Agentscited by 7
  16. 2024Adversarial Style Augmentation via Large Language Model for Robust Fake News Detectioncited by 5
  17. 2024Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generationcited by 5
  18. 2024Gradual Vigilance and Interval Communication: Enhancing Value Alignment in Multi-Agent Debatescited by 4
  19. 2025Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selectioncited by 4
  20. 2024Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspectivecited by 3
  21. 2024ChatDyn: Language-Driven Multi-Actor Dynamics Generation in Street Scenescited by 3
  22. 2025Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventionscited by 3
  23. 2025QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulationcited by 3
  24. 2024Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Modelscited by 2
  25. 2025A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMscited by 2
  26. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  27. 2025Social Welfare Function Leaderboard: When LLM Agents Allocate Social Welfarecited by 2
  28. 2025Debiasing International Attitudes: LLM Agents for Simulating US-China Perception Changescited by 1
  29. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 1
  30. 2025Towards Ethical Multi-Agent Systems of Large Language Models: A Mechanistic Interpretability Perspectivecited by 1
  31. 2026Simple Role Assignment is Extraordinarily Effective for Safety Alignmentcited by 1
  32. 2025Epitome: Pioneering an Experimental Platform for AI-Social Science Integrationcited by 0
  33. 2026Toward a Theory of Value in AI Alignmentcited by 0
  34. 2026Statutory AI: Aligning Large Language Models With Legal Normscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
  2. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  3. 2024Benchmarking Distributional Alignment of Large Language Models
  4. 2023Large Language Model Alignment: A Survey
  5. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoning
  6. 2024Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks
  7. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  8. 2026LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations