mlsanityData to 14 Oct 2026

Inference-time Alignment in Continuous Space

arXiv 2505.20081 ↗First posted 26 May 2025cs.CL cs.AIcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models with human feedback at inference time has received increasing attention due to its flexibility. Existing methods rely on generating multiple responses from the base policy for search using a reward model, which can be considered as searching in a discrete response space. However, these methods struggle to explore informative candidates when the base policy is weak or the candidate set is small, resulting in limited effectiveness. In this paper, to address this problem, we propose Simple Energy Adaptation (SEA), a simple yet effective algorithm for inference-time alignment. In contrast to expensive search over the discrete space, SEA directly adapts original responses from the base policy toward the optimal one via gradient-based sampling in continuous latent space. Specifically, SEA formulates inference as an iterative optimization procedure on an energy function over actions in the continuous space defined by the optimal policy, enabling simple and effective alignment. For instance, despite its simplicity, SEA outperforms the second-best baseline with a relative improvement of up to 77.51% on AdvBench and 16.36% on MATH. Our code is publicly available at https://github.com/yuanyige/sea

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 64 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

54 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025On a Connection Between Imitation Learning and RLHF7 Mar
  2. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters2 Feb
  3. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  4. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective14 Oct
  5. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  6. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  8. 2024Spectral Editing of Activations for Large Language Model Alignment15 May
  9. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  10. 2024Decoding-time Realignment of Language Models5 Feb
  11. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  12. 2024Towards Efficient Exact Optimization of Language Model Alignment1 Feb
2 more
  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
28 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  10. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  11. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  12. 2021How to Train Your Energy-Based Modelscited by 274
  13. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  14. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  15. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  16. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  17. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  18. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 161
  19. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  20. 2023Controlled Decoding from Language Modelscited by 135
  21. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  22. 2020Residual Energy-Based Models for Text Generationcited by 95
  23. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  24. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  25. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  26. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  27. 2023TEA: Test-time Energy Adaptationcited by 25
  28. 2024MITA: Bridging the Gap between Model and Data for Test-time Adaptationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  2. 2026nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Spacecited by 5
  3. 2026Inference-time Alignment via Sparse Junction Steeringcited by 2
  4. 2026LatentRevise: Learning from Zero-Hit Reasoningcited by 2
  5. 2025Alignment-Aware Decodingcited by 1
  6. 2026Gradient-Guided Reward Optimization for Inference-time Alignmentcited by 0
  7. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Timecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment
  2. 2026Towards a theory of inference-time alignment with unknown rewards
  3. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  4. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  5. 2025AdaBoN: Adaptive Best-of-N Alignment
  6. 2024Inference Time Alignment with Reward-Guided Tree Search
  7. 2024InfAlign: Inference-aware language model alignment
  8. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner