mlsanityData to 14 Oct 2026

Aligning Deep Implicit Preferences by Learning to Reason Defensively

arXiv 2510.11194 ↗First posted 13 Oct 2025cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Personalized alignment is crucial for enabling Large Language Models (LLMs) to engage effectively in user-centric interactions. However, current methods face a dual challenge: they fail to infer users' deep implicit preferences (including unstated goals, semantic context and risk tolerances), and they lack the defensive reasoning required to navigate real-world ambiguity. This cognitive gap leads to responses that are superficial, brittle and short-sighted. To address this, we propose Critique-Driven Reasoning Alignment (CDRA), which reframes alignment from a scalar reward-matching task into a structured reasoning process. First, to bridge the preference inference gap, we introduce the DeepPref benchmark. This dataset, comprising 3000 preference-query pairs across 20 topics, is curated by simulating a multi-faceted cognitive council that produces critique-annotated reasoning chains to deconstruct query semantics and reveal latent risks. Second, to instill defensive reasoning, we introduce the Personalized Generative Process Reward Model (Pers-GenPRM), which frames reward modeling as a personalized reasoning task. It generates a critique chain to evaluate a response's alignment with user preferences before outputting a final score based on this rationale. Ultimately, this interpretable, structured reward signal guides policy model through Critique-Driven Policy Alignment, a process-level online reinforcement learning algorithm integrating both numerical and natural language feedback. Experiments demonstrate that CDRA excels at discovering and aligning with users' true preferences while executing robust reasoning. Our code and dataset are available at https://github.com/Zephyrian-Hugh/Deep-pref.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 47 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

28 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  3. 2023Let's Verify Step by Step31 May
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  10. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  12. 2022Self-critiquing models for assisting human evaluatorscited by 380
12 more
  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  2. 2023AI Alignment: A Comprehensive Surveycited by 263
  3. 2022Defining and Characterizing Reward Hackingcited by 203
  4. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  5. 2024LLM Critics Help Catch LLM Bugscited by 147
  6. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  7. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 63
  8. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  9. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 35
  10. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
  11. 2024Dissecting Human and LLM Preferencescited by 17
  12. 2024PersLLM: A Personified Training Approach for Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulationcited by 0
  2. 2026Calibrating LLM Explanations to Human Decisions with Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
  2. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment
  3. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  4. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  5. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
  6. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signals
  7. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  8. 2025GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs