mlsanityData to 14 Oct 2026

Training Socially Aligned Language Models on Simulated Social Interactions

arXiv 2305.16960 ↗First posted 26 May 2023cs.CL cs.AI cs.CYcited by 78

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Social alignment in AI systems aims to ensure that these models behave according to established societal values. However, unlike humans, who derive consensus on value judgments through social interaction, current language models (LMs) are trained to rigidly replicate their training corpus in isolation, leading to subpar generalization in unfamiliar scenarios and vulnerability to adversarial attacks. This work presents a novel training paradigm that permits LMs to learn from simulated social interactions. In comparison to existing methodologies, our approach is considerably more scalable and efficient, demonstrating superior performance in alignment benchmarks and human evaluations. This paradigm shift in the training of LMs brings us a step closer to developing AI systems that can robustly and accurately reflect societal norms and values.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 79 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

54 of 79 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Training Language Models with Language Feedback at Scale28 Mar
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2020Aligning AI With Shared Human Values5 Aug

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  9. 2022Large Language Models Can Self-Improvecited by 664
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  11. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  13. 2020Artificial Intelligence, Values and Alignmentcited by 448
  14. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  15. 2018AI safety via debatecited by 400
  16. 2022Self-critiquing models for assisting human evaluatorscited by 380
  17. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  18. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  19. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  20. 2022Generating Sequences by Learning to Self-Correctcited by 277
  21. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  22. 2023The False Promise of Imitating Proprietary LLMscited by 263
  23. 2021Alignment of Language Agentscited by 168
  24. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  25. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  26. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  27. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  28. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  29. 2022Language Models as Agent Modelscited by 132
  30. 2017AI Safety Gridworldscited by 110
  31. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  32. 2022Language Model Cascadescited by 107
  33. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  34. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 93
  35. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  36. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  37. 2022CommunityLM: Probing Partisan Worldviews from Language Modelscited by 36
  38. 2020Avoiding Side Effects By Considering Future Taskscited by 32

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 33; 2025: 29; 2026: 9.

78 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  8. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 186
  9. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 132
  10. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  11. 2024LLM Voting: Human Choices and AI Collective Decision Makingcited by 48
  12. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
38 more
  1. 2024Emergence of Social Norms in Generative Agent Societies: Principles and Architecturecited by 41
  2. 2024Moral Alignment for LLM Agentscited by 39
  3. 2025Finetuning LLMs for Human Behavior Prediction in Social Science Experimentscited by 39
  4. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attackscited by 34
  5. 2023Aligning Large Language Models with Human: A Surveycited by 32
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  7. 2024Simulating Human-like Daily Activities with Desire-driven Autonomycited by 30
  8. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29
  9. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  10. 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberationcited by 26
  11. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  12. 2025MDTeamGPT: A Self-Evolving LLM-based Multi-Agent Framework for Multi-Disciplinary Team Medical Consultationcited by 26
  13. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 25
  14. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  15. 2024Autonomous Agents for Collaborative Task under Information Asymmetrycited by 16
  16. 2024MorphAgent: Empowering Agents through Self-Evolving Profiles and Decentralized Collaborationcited by 16
  17. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  18. 2025Internal Value Alignment in Large Language Models through Controlled Value Vector Activationcited by 16
  19. 2024An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Makingcited by 14
  20. 2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Executioncited by 14
  21. 2025Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviorscited by 13
  22. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 13
  23. 2024OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Modelscited by 12
  24. 2023Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulationscited by 11
  25. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  26. 2025MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attackscited by 11
  27. 2024Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoningcited by 10
  28. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  29. 2024COPR: Continual Human Preference Learning via Optimal Policy Regularizationcited by 9
  30. 2024INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulnesscited by 8
  31. 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
  32. 2025Modeling Earth-Scale Human-Like Societies with One Billion Agentscited by 8
  33. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Modelscited by 5
  34. 2025Trustworthy AI: Safety, Bias, and Privacy -- A Surveycited by 5
  35. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  36. 2025Learning to Make Friends: Coaching LLM Agents toward Emergent Social Tiescited by 5
  37. 2025Ella: Embodied Social Agents with Lifelong Memorycited by 4
  38. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoning
  2. 2026Social-R1: Towards Human-like Social Reasoning in LLMs
  3. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models
  4. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviors
  5. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
  6. 2023Large Language Model Alignment: A Survey
  7. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  8. 2026HAL: Inducing Human-likeness in LLMs with Alignment