# Value Alignment Verification

arXiv: https://arxiv.org/abs/2012.01557 · HTML: https://mlsanity.com/paper/2012.01557/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 20 of 20)

- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL](https://mlsanity.com/paper/2206.00695/) (2022)
- [High-Dimension Human Value Representation in Large Language Models](https://mlsanity.com/paper/2404.07900/) (2024)
- [Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning](https://mlsanity.com/paper/2209.10974/) (2022)
- [Preference Poisoning Attacks on Reward Model Learning](https://mlsanity.com/paper/2402.01920/) (2024)
- [Contextual Moral Value Alignment Through Context-Based Aggregation](https://mlsanity.com/paper/2403.12805/) (2024)
- [Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners](https://mlsanity.com/paper/2503.05996/) (2025)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Unpacking the Ethical Value Alignment in Big Models](https://mlsanity.com/paper/2310.17551/) (2023)
- [Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment](https://mlsanity.com/paper/2406.04231/) (2024)
- [Heterogeneous Value Alignment Evaluation for Large Language Models](https://mlsanity.com/paper/2305.17147/) (2023)
- [MAPLE: A Framework for Active Preference Learning Guided by Large Language Models](https://mlsanity.com/paper/2412.07207/) (2024)
- [Towards Regulatable AI Systems: Technical Gaps and Policy Opportunities](https://mlsanity.com/paper/2306.12609/) (2023)
- [X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking](https://mlsanity.com/paper/2606.22278/) (2026)
- [Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning](https://mlsanity.com/paper/2607.08647/) (2026)
- [Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?](https://mlsanity.com/paper/2306.13004/) (2023)
- [Dynamic value alignment through preference aggregation of multiple objectives](https://mlsanity.com/paper/2310.05871/) (2023)
- [HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kong](https://mlsanity.com/paper/2507.11502/) (2025)
- [Rethinking How AI Embeds and Adapts to Human Values: Challenges and Opportunities](https://mlsanity.com/paper/2508.17104/) (2025)
- [The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning](https://mlsanity.com/paper/2601.16906/) (2026)

## Abstract (arXiv)

As humans interact with autonomous agents to perform increasingly complicated, potentially risky tasks, it is important to be able to efficiently evaluate an agent's performance and correctness. In this paper we formalize and theoretically analyze the problem of efficient value alignment verification: how to efficiently test whether the behavior of another agent is aligned with a human's values. The goal is to construct a kind of "driver's test" that a human can give to any agent which will verify value alignment via a minimal number of queries. We study alignment verification problems with both idealized humans that have an explicit reward function as well as problems where they have implicit values. We analyze verification of exact value alignment for rational agents and propose and analyze heuristic and approximate value alignment verification tests in a wide range of gridworlds and a continuous autonomous driving domain. Finally, we prove that there exist sufficient conditions such that we can verify exact and approximate alignment across an infinite set of test environments via a constant-query-complexity alignment test.