mlsanityData to 7 Oct 2026

Concept Alignment as a Prerequisite for Value Alignment

arXiv 2310.20059 ↗First posted 30 Oct 2023cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Value alignment is essential for building AI systems that can safely and reliably interact with people. However, what a person values -- and is even capable of valuing -- depends on the concepts that they are currently using to understand and evaluate what happens in the world. The dependence of values on concepts means that concept alignment is a prerequisite for value alignment -- agents need to align their representation of a situation with that of humans in order to successfully align their values. Here, we formally analyze the concept alignment problem in the inverse reinforcement learning setting, show how neglecting concept alignment can lead to systematic value mis-alignment, and describe an approach that helps minimize such failure modes by jointly reasoning about a person's concepts and values. Additionally, we report experimental results with human participants showing that humans reason about the concepts used by an agent when acting intentionally, in line with our joint reasoning model.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 10 of its 22 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

10 of 22 references.

References · 10

  1. 2016Cooperative Inverse Reinforcement Learningcited by 298
  2. 2020Online Bayesian Goal Inference for Boundedly-Rational Planning Agentscited by 67
  3. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  4. 2021People construct simplified mental representations to plancited by 45
  5. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  6. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  7. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  8. 2021Human irrationality: both bad and good for reward inferencecited by 25
  9. 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
  10. 2021Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mindcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  2. 2023Learning Human-like Representations to Enable Learning Human Valuescited by 7
  3. 2024Representational Alignment Supports Effective Machine Teachingcited by 3
  4. 2025Estimating cognitive biases with attention-aware inverse planningcited by 2
  5. 2026Investigating Concept Alignment Using Implausible Category Memberscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Concept Alignment
  2. 2025The Specification Trap: Why Static Value Alignment Alone Is Insufficient for Robust Alignment
  3. 2026Toward a Theory of Value in AI Alignment
  4. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  5. 2024Value Alignment and Trust in Human-Robot Interaction: Insights from Simulation and User Study
  6. 2021Value alignment: a formal approach
  7. 2024Position: Towards Bidirectional Human-AI Alignment
  8. 2024Democratizing Reward Design for Personal and Representative Value-Alignment