mlsanityData to 7 Oct 2026

Roadmap on Incentive Compatibility for AI Alignment and Governance in Sociotechnical Systems

arXiv 2402.12907 ↗First posted 20 Feb 2024cs.AI cs.CY cs.GTcited by 6

Not on the map. Why?

Abstract (arXiv)

The burgeoning integration of artificial intelligence (AI) into human society brings forth significant implications for societal governance and safety. While considerable strides have been made in addressing AI alignment challenges, existing methodologies primarily focus on technical facets, often neglecting the intricate sociotechnical nature of AI systems, which can lead to a misalignment between the development and deployment contexts. To this end, we posit a new problem worth exploring: Incentive Compatibility Sociotechnical Alignment Problem (ICSAP). We hope this can call for more researchers to explore how to leverage the principles of Incentive Compatibility (IC) from game theory to bridge the gap between technical and societal components to maintain AI consensus with human societies in different contexts. We further discuss three classical game problems for achieving IC: mechanism design, contract theory, and Bayesian persuasion, in addressing the perspectives, potentials, and challenges of solving ICSAP, and provide preliminary implementation conceptions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 84 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

30 of 84 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  11. 2018AI safety via debatecited by 384
  12. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
18 more
  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  2. 2023AI Alignment: A Comprehensive Surveycited by 256
  3. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  4. 2020Open Problems in Cooperative AIcited by 167
  5. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  6. 2018Supervising strong learners by amplifying weak expertscited by 139
  7. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  8. 2020The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policiescited by 75
  9. 2022Mechanistic Mode Connectivitycited by 50
  10. 2018Fairness Behind a Veil of Ignorance: A Welfare Analysis for Automated Decision Makingcited by 46
  11. 2021Robustness and Generalization via Generative Adversarial Trainingcited by 20
  12. 2022Multi-Agent Policy Transfer via Task Relationship Modelingcited by 14
  13. 2023Symmetry-Aware Robot Design with Structured Subgroupscited by 13
  14. 2021Bayesian Persuasion in Sequential Decision-Makingcited by 12
  15. 2022Non-Linear Coordination Graphscited by 10
  16. 2024Multi-Sender Persuasion: A Computational Perspectivecited by 9
  17. 2021Axes for Sociotechnical Inquiry in AI Researchcited by 5
  18. 2015Mechanism Design for Fair Allocationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  2. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  3. 2024Efficient Model-agnostic Alignment via Bayesian Persuasioncited by 8
  4. 2026PILA: Plug-and-Play Insertion for LLM-native Advertisingcited by 2
  5. 2025Make an Offer They Can't Refuse: Grounding Bayesian Persuasion in Real-World Dialogues without Pre-Commitmentcited by 1
  6. 2026Agentic AI, Retrieval-Augmented Generation, and the Institutional Turn: Legal Architectures and Financial Governance in the Age of Distributional AGIcited by 0