mlsanityData to 14 Oct 2026

KTO: Model Alignment as Prospect Theoretic Optimization

arXiv 2402.01306 ↗First posted 2 Feb 2024cs.LG cs.AIcited by 264

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Kahneman & Tversky's textitprospect theory tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases -- the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them belonging to a family of loss functions that we call textithuman-aware losses (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach KTO, and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B, despite only learning from a binary signal of whether an output is desirable. More broadly, our work suggests that there is no one HALO that is universally superior; the best loss depends on the inductive biases most appropriate for a given setting, an oft-overlooked consideration.

Key facts. Cited by 264 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 57 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

40 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023Mistral 7Bcited by 3,164
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
26 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2017Neural Collaborative Filteringcited by 1,111
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  9. 2024Self-Rewarding Language Modelscited by 681
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  13. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  14. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  15. 2023Pretraining Language Models with Human Preferencescited by 247
  16. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  17. 2023Nash Learning from Human Feedbackcited by 224
  18. 2023Fine-tuning Language Models for Factualitycited by 221
  19. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  20. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  21. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  22. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  23. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  24. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  25. 2021Human irrationality: both bad and good for reward inferencecited by 25
  26. 2019Interpretable Modelling of Driving Behaviors in Interactive Driving Scenarios based on Cumulative Prospect Theorycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 100; 2026: 138.

264 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  3. 2024Generative Representational Instruction Tuningcited by 233
  4. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  8. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  9. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  10. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  11. 2025Reinforcement Learning from Human Feedbackcited by 58
  12. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
38 more
  1. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  2. 2025Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modelingcited by 45
  3. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  4. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Modelscited by 39
  5. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
  6. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  7. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  8. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 30
  9. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  10. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  11. 2025Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Modelscited by 27
  12. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
  13. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 26
  14. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  15. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  16. 2025Language Models that Think, Chat Bettercited by 24
  17. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  18. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 23
  19. 2025BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillationcited by 23
  20. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 23
  21. 2025DyFlow: Dynamic Workflow Framework for Agentic Reasoningcited by 23
  22. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  23. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
  24. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  25. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  26. 2025Robust Reward Modeling via Causal Rubricscited by 19
  27. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  28. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  29. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  30. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  31. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  32. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  33. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  34. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  35. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoningcited by 14
  36. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  37. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoningcited by 12
  38. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11

50 most cited of 264 All 264 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Humanline: Online Alignment as Perceptual Loss
  2. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
  3. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  4. 2025Evaluating and Aligning Human Economic Risk Preferences in LLMs
  5. 2026Probing Outcome-Level Resemblance and Mechanism-Level Alignment in LLM Risk Decisions: Evidence from the St. Petersburg Game
  6. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
  7. 2021Uncertain Decisions Facilitate Better Preference Learning
  8. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model