mlsanityData to 14 Oct 2026

Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry

arXiv 2610.20350 ↗First posted 14 Oct 2026cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Preference optimization ranks final answers and trajectory RL maximizes aggregate returns, yet neither credits the local change that turns a flawed reasoning state into a better one. We introduce Lifted State Policy Optimization (LSPO), which establishes this finite, cost-adjusted transition as the primary unit of learning. Each textual answer is augmented with a learned continuous coordinate, forming a lifted state that an energy function evaluates and orders. A shaped reward drives energy descent net of edit and step costs, and an internalization loss absorbs selected low-energy answers into the direct generator. On Qwen3.5-27B, the resulting single-pass policy reaches 71.8% aggregate accuracy across mathematics, science, code, and logic benchmarks, matching test-time reflection at 71.9% with 28% lower latency and 25.8% fewer tokens. The pattern holds across dense and mixture-of-experts backbones, and matched controls confirm that transition credit and the lifted coordinate each contribute, as does answer internalization. Our code is available at https://github.com/Venomrko/lspo.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 82 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

62 of 82 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation10 Feb
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs26 Jun
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 59

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2017On Calibration of Modern Neural Networkscited by 4,960
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  13. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  14. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  15. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  16. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  17. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  18. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  19. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  20. 2024Self-Rewarding Language Modelscited by 681
  21. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  22. 2025Muon is Scalable for LLM Trainingcited by 428
  23. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  24. 2024Measuring short-form factuality in large language modelscited by 373
  25. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  26. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  27. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  28. 2023Do Large Language Models Know What They Don't Know?cited by 243
  29. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  30. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  31. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  32. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
  33. 2025Self-rewarding correction for mathematical reasoningcited by 70
  34. 2025Reinforcement Learning from Human Feedbackcited by 58
  35. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
  36. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  37. 2025A Survey of Direct Preference Optimizationcited by 36
  38. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 35

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy Optimization Prefers The Path of Least Resistance
  2. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
  3. 2026Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models
  4. 2026VASPO: Self-Distillation with Privileged Value Estimation
  5. 2026Teach to Learn: Hint Annealing for Self-improving LLM Reasoning
  6. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
  7. 2026Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces
  8. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation