mlsanityData to 14 Oct 2026

InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

arXiv 2512.23126 ↗First posted 29 Dec 2025cs.AI cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) and its variants have become standard for aligning Large Language Models due to their simplicity and offline stability. However, we identify two fundamental limitations. First, the optimal policy depends on arbitrary modeling choices (scalarization function, reference policy), yielding behavior reflecting parameterization artifacts rather than true preferences. Second, treating response generation in isolation fails to leverage comparative information in pairwise data, leaving the model's capacity for intrinsic self-reflection untapped. To address it, we propose Intrinsic Self-reflective Preference Optimization (InSPO), deriving a globally optimal policy conditioning on both context and alternative responses. We prove this formulation superior to DPO/RLHF while guaranteeing invariance to scalarization and reference choices. InSPO serves as a plug-and-play enhancement without architectural changes or inference overhead. Experiments demonstrate consistent improvements in win rates and length-controlled metrics, validating that unlocking self-reflection yields more robust, human-aligned LLMs. Our Code is available at https://github.com/Skylanding/InSPO.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 38 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

28 of 38 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Token-level Direct Preference Optimization18 Apr
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  11. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  12. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
7 more
  1. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  2. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  3. 2025Network Diffuser for Placing-Scheduling Service Function Chains with Inverse Demonstrationcited by 11
  4. 2025Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimizationcited by 9
  5. 2025A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Realitycited by 8
  6. 2024Mixed Text Recognition with Efficient Parameter Fine-Tuning and Transformercited by 4
  7. 2025Tunable Leg Stiffness in a Monopedal Hopper for Energy-Efficient Vertical Hopping Across Varying Ground Profilescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  2. 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agentscited by 12
  3. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 10
  4. 2026Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimizationcited by 8
  5. 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 7
  6. 2026Recursive Self-Evolving Agents via Held-Out Selectioncited by 7
  7. 2026PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Modelscited by 5
  8. 2026MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitationcited by 4
  9. 2026ACDZero: MCTS Agent for Mastering Automated Cyber Defensecited by 2
  10. 2026When Should Service Agents Reconsider? Difficulty-Routed Control in Customer-Service Operationscited by 2
  11. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025SGPO: Self-Generated Preference Optimization based on Self-Improver
  2. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  3. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  4. 2025A Survey of Direct Preference Optimization
  5. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward
  6. 2024Understanding Reference Policies in Direct Preference Optimization
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation