mlsanityData to 14 Oct 2026

Humanline: Online Alignment as Perceptual Loss

arXiv 2509.24207 ↗First posted 29 Sep 2025cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Online alignment (e.g., GRPO) is generally more performant than offline alignment (e.g., DPO) -- but why? Drawing on prospect theory from behavioral economics, we propose a human-centric explanation. We prove that online on-policy sampling better approximates the human-perceived distribution of what the model can produce, and PPO/GRPO-style clipping -- originally introduced to just stabilize training -- recovers a perceptual bias in how humans perceive probability. In this sense, PPO/GRPO act as perceptual losses already. Our theory further suggests that the online/offline dichotomy is itself incidental to maximizing human utility, since we can achieve the same effect by selectively training on any data in a manner that mimics human perception, rather than restricting ourselves to online on-policy data. Doing so would allow us to post-train more quickly, cheaply, and flexibly without sacrificing performance. To this end, we propose a design pattern that explicitly incorporates perceptual distortions of probability into objectives like DPO/KTO/GRPO, creating humanline variants of them. Surprisingly, we find that these humanline variants, even when trained with offline off-policy data, can match the performance of their online counterparts (on both verifiable and unverifiable tasks) while running up to 6x faster.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 84 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

64 of 84 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025Reinforcing General Reasoning without Verifiers27 May
  4. 2025Learning to Reason without External Rewards26 May
  5. 2025INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning12 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  12. 2023Let's Verify Step by Step31 May
3 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
37 more
  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  7. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  9. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  12. 2024Iterative Reasoning Preference Optimizationcited by 246
  13. 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 234
  14. 2023Nash Learning from Human Feedbackcited by 224
  15. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  16. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  17. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  18. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  19. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 114
  20. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  21. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  22. 2024Critique-out-Loud Reward Modelscited by 106
  23. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  24. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  25. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 77
  26. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  27. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  28. 2024Learn Your Reference Model for Real Good Alignmentcited by 57
  29. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  30. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
  31. 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 45
  32. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  33. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  34. 2021Human irrationality: both bad and good for reward inferencecited by 25
  35. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  36. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  37. 2019Interpretable Modelling of Driving Behaviors in Interactive Driving Scenarios based on Cumulative Prospect Theorycited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024KTO: Model Alignment as Prospect Theoretic Optimization
  2. 2025Direct Alignment with Heterogeneous Preferences
  3. 2026Probing Outcome-Level Resemblance and Mechanism-Level Alignment in LLM Risk Decisions: Evidence from the St. Petersburg Game
  4. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  5. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
  6. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  7. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation
  8. 2025Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability