# Learning Human-like Representations to Enable Learning Human Values

arXiv: https://arxiv.org/abs/2312.14106 · HTML: https://mlsanity.com/paper/2312.14106/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 79 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2010.08240/) (2020)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](https://mlsanity.com/paper/2104.06979/) (2021)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [What Would Jiminy Cricket Do? Towards Agents That Behave Morally](https://mlsanity.com/paper/2110.13136/) (2021)
- [The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes](https://mlsanity.com/paper/2012.14210/) (2020)
- [Safe Exploration of State and Action Spaces in Reinforcement Learning](https://mlsanity.com/paper/1402.0560/) (2014)
- [Improving neural network representations using human similarity judgments](https://mlsanity.com/paper/2306.04507/) (2023)
- [Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation](https://mlsanity.com/paper/2311.00306/) (2023)
- [Alignment with human representations supports robust few-shot learning](https://mlsanity.com/paper/2301.11990/) (2023)
- [Words are all you need? Language as an approximation for human similarity judgments](https://mlsanity.com/paper/2206.04105/) (2022)
- [Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior](https://mlsanity.com/paper/2104.09469/) (2021)
- [Predicting Human Similarity Judgments Using Large Language Models](https://mlsanity.com/paper/2202.04728/) (2022)
- [Concept Alignment as a Prerequisite for Value Alignment](https://mlsanity.com/paper/2310.20059/) (2023)

## Built on by (most cited first, 7 of 7)

- [Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing](https://mlsanity.com/paper/2406.14230/) (2024)
- [Representational Alignment Supports Effective Machine Teaching](https://mlsanity.com/paper/2406.04302/) (2024)
- [Toward Human-AI Alignment in Large-Scale Multi-Player Games](https://mlsanity.com/paper/2402.03575/) (2024)
- [Toward Cultural Interpretability: A Linguistic Anthropological Framework for Describing and Evaluating Large Language Models (LLMs)](https://mlsanity.com/paper/2411.05200/) (2024)
- [Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2602.08835/) (2026)
- [Towards Multi-Agent-Simulation-Based Community Note Evaluation](https://mlsanity.com/paper/2606.18268/) (2026)
- [Evaluating Persistent Calibration under Evolving Model Knowledge](https://mlsanity.com/paper/2609.38797/) (2026)

## Abstract (arXiv)

How can we build AI systems that can learn any set of individual human values both quickly and safely, avoiding causing harm or violating societal standards for acceptable behavior during the learning process? We explore the effects of representational alignment between humans and AI agents on learning human values. Making AI systems learn human-like representations of the world has many known benefits, including improving generalization, robustness to domain shifts, and few-shot learning performance. We demonstrate that this kind of representational alignment can also support safely learning and exploring human values in the context of personalization. We begin with a theoretical prediction, show that it applies to learning human morality judgments, then show that our results generalize to ten different aspects of human values -- including ethics, honesty, and fairness -- training AI agents on each set of values in a multi-armed bandit setting, where rewards reflect human value judgments over the chosen action. Using a set of textual action descriptions, we collect value judgments from humans, as well as similarity judgments from both humans and multiple language models, and demonstrate that representational alignment enables both safe exploration and improved generalization when learning human values.