# Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation

arXiv: https://arxiv.org/abs/2402.05699 · HTML: https://mlsanity.com/paper/2402.05699/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 42 of its 65 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 42)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)

## Built on by (most cited first, 30 of 46)

- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents](https://mlsanity.com/paper/2412.13178/) (2024)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Self-Evolving Multi-Agent Collaboration Networks for Software Development](https://mlsanity.com/paper/2410.16946/) (2024)
- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [Internal Consistency and Self-Feedback in Large Language Models: A Survey](https://mlsanity.com/paper/2407.14507/) (2024)
- [Alignment of Diffusion Models: Fundamentals, Challenges, and Future](https://mlsanity.com/paper/2409.07253/) (2024)
- [From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users](https://mlsanity.com/paper/2508.17281/) (2025)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025)
- [We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems](https://mlsanity.com/paper/2506.13666/) (2025)
- [ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering](https://mlsanity.com/paper/2505.23723/) (2025)
- [Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation](https://mlsanity.com/paper/2410.14251/) (2024)
- [Sotopia-RL: Reward Design for Social Intelligence](https://mlsanity.com/paper/2508.03905/) (2025)
- [MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization](https://mlsanity.com/paper/2410.07672/) (2024)
- [Plurals: A System for Guiding LLMs Via Simulated Social Ensembles](https://mlsanity.com/paper/2409.17213/) (2024)
- [Training Language Models to Win Debates with Self-Play Improves Judge Accuracy](https://mlsanity.com/paper/2409.16636/) (2024)
- [MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2505.16988/) (2025)
- [MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training](https://mlsanity.com/paper/2502.11541/) (2025)
- [SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulation](https://mlsanity.com/paper/2504.16122/) (2025)
- [ProgressGym: Alignment with a Millennium of Moral Progress](https://mlsanity.com/paper/2406.20087/) (2024)
- [SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat](https://mlsanity.com/paper/2506.04721/) (2025)
- [PathVLM-R1: A Reinforcement Learning-Driven Reasoning Model for Pathology Visual-Language Tasks](https://mlsanity.com/paper/2504.09258/) (2025)
- [Infusing Theory of Mind into Socially Intelligent LLM Agents](https://mlsanity.com/paper/2509.22887/) (2025)
- [Adversarial Style Augmentation via Large Language Model for Robust Fake News Detection](https://mlsanity.com/paper/2406.11260/) (2024)
- [Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generation](https://mlsanity.com/paper/2412.05342/) (2024)
- [Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection](https://mlsanity.com/paper/2504.20644/) (2025)

## Abstract (arXiv)

Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse. Drawing from the sociological insight that acknowledging all parties' concerns is a key factor in shaping human values, this paper proposes a novel direction to align LLMs by themselves: social scene simulation. To achieve this, we present MATRIX, a novel social scene simulator that emulates realistic scenes around a user's input query, enabling the LLM to take social consequences into account before responding. MATRIX serves as a virtual rehearsal space, akin to a Monopolylogue, where the LLM performs diverse roles related to the query and practice by itself. To inject this alignment, we fine-tune the LLM with MATRIX-simulated data, ensuring adherence to human values without compromising inference speed. We theoretically show that the LLM with MATRIX outperforms Constitutional AI under mild assumptions. Finally, extensive experiments validate that our method outperforms over 10 baselines across 4 benchmarks. As evidenced by 875 user ratings, our tuned 13B-size LLM exceeds GPT-4 in aligning with human values. See our project page at https://shuotang123.github.io/MATRIX.