mlsanityData to 7 Oct 2026

Social Contract AI: Aligning AI Assistants with Implicit Group Norms

arXiv 2310.17769 ↗First posted 26 Oct 2023cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

We explore the idea of aligning an AI assistant by inverting a model of users' (unknown) preferences from observed interactions. To validate our proposal, we run proof-of-concept simulations in the economic ultimatum game, formalizing user preferences as policies that guide the actions of simulated players. We find that the AI assistant accurately aligns its behavior to match standard policies from the economic literature (e.g., selfish, altruistic). However, the assistant's learned policies lack robustness and exhibit limited generalization in an out-of-distribution setting when confronted with a currency (e.g., grams of medicine) that was not included in the assistant's training distribution. Additionally, we find that when there is inconsistency in the relationship between language use and an unknown policy (e.g., an altruistic policy combined with rude language), the assistant's learning of the policy is slowed. Overall, our preliminary results suggest that developing simulation frameworks in which AI assistants need to infer preferences from diverse users can provide a valuable approach for studying practical alignment questions.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 78 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

35 of 78 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2023Large Language Models as Optimizerscited by 830
  11. 2017A Tutorial on Thompson Samplingcited by 420
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
23 more
  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  2. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  3. 2015Contextual Markov Decision Processescited by 213
  4. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  5. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  6. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  7. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  8. 2021On the Expressivity of Markov Rewardcited by 71
  9. 2019Meta-learning of Sequential Strategiescited by 68
  10. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  11. 2023Strategic Reasoning with Language Modelscited by 64
  12. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  13. 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
  14. 2017Markov Decision Processes with Continuous Side Informationcited by 54
  15. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  16. 2017Efficient Parallel Methods for Deep Reinforcement Learningcited by 32
  17. 2019Theory of Minds: Understanding Behavior in Groups Through Inverse Planningcited by 30
  18. 2022Task Ambiguity in Humans and Language Modelscited by 27
  19. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  20. 2018Coordinated Exploration in Concurrent Reinforcement Learningcited by 17
  21. 2019Inverse Reinforcement Learning in Contextual MDPscited by 12
  22. 2018Scalable Coordinated Exploration in Concurrent Reinforcement Learningcited by 10
  23. 2016Posterior Sampling for Reinforcement Learning Without Episodescited by 6

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 3.

9 papers.

  1. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  2. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  3. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  4. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 18
  5. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  6. 2024Engagement-Driven Content Generation with Large Language Modelscited by 5
  7. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5
  8. 2024Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Modelscited by 4
  9. 2025Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?cited by 1