# Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignment

arXiv: https://arxiv.org/abs/2509.04445 · HTML: https://mlsanity.com/paper/2509.04445/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 85 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI](https://mlsanity.com/paper/2010.07487/) (2020)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)
- [Axioms for AI Alignment from Human Feedback](https://mlsanity.com/paper/2405.14758/) (2024)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [What are you optimizing for? Aligning Recommender Systems with Human Values](https://mlsanity.com/paper/2107.10939/) (2021)
- [Cognitive Model Priors for Predicting Human Decisions](https://mlsanity.com/paper/1905.09397/) (2019)
- [A Survey on Human Preference Learning for Large Language Models](https://mlsanity.com/paper/2406.11191/) (2024)
- [Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making](https://mlsanity.com/paper/2102.00625/) (2021)
- [Optimal Design for Human Preference Elicitation](https://mlsanity.com/paper/2404.13895/) (2024)
- [A Computational Model of Commonsense Moral Decision Making](https://mlsanity.com/paper/1801.04346/) (2018)
- [Predicting human decision making in psychological tasks with recurrent neural networks](https://mlsanity.com/paper/2010.11413/) (2020)
- [Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions](https://mlsanity.com/paper/2503.00940/) (2025)
- [Deploying a Robust Active Preference Elicitation Algorithm on MTurk: Experiment Design, Interface, and Evaluation for COVID-19 Patient Prioritization](https://mlsanity.com/paper/2306.04061/) (2023)
- [Value Preferences Estimation and Disambiguation in Hybrid Participatory Systems](https://mlsanity.com/paper/2402.16751/) (2024)
- [On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods](https://mlsanity.com/paper/2408.02862/) (2024)

## Built on by (most cited first, 3 of 3)

- [FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2510.04040/) (2025)
- [Internal Pluralism and the Limits of Pairwise Comparisons](https://mlsanity.com/paper/2607.02672/) (2026)
- [Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning](https://mlsanity.com/paper/2608.12372/) (2026)

## Abstract (arXiv)

Recent AI trends seek to align AI models to learned human-centric objectives, such as personal preferences, utility, or societal values. Using standard preference elicitation methods, researchers and practitioners build models of human decisions and judgments, to which AI models are aligned. However, standard elicitation methods often fail to capture the cognitive processes behind human decision making, such as heuristics or simplifying structured thought patterns. To address this failure, we take an axiomatic approach to learning cognitively faithful decision processes from pairwise comparisons. Building on the literature analyzing cognitive processes that shape human decision-making, we derive a model class in which features are first processed with learned rules, then aggregated via a fixed rule, such as the Bradley-Terry rule, to produce a decision. This structured processing of information ensures that such models are realistic and feasible candidates to represent underlying human decision-making processes. We demonstrate the efficacy of this modeling approach by learning interpretable models of human decision making in a kidney allocation task, and show that our proposed models match or surpass the accuracy of prior models of human pairwise decision-making.