mlsanityData to 7 Oct 2026

Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback

arXiv 2501.12895 ↗First posted 22 Jan 2025cs.CLcited by 41

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) demonstrate impressive performance but lack the flexibility to adapt to human preferences quickly without retraining. In this work, we introduce Test-time Preference Optimization (TPO), a framework that aligns LLM outputs with human preferences during inference, removing the need to update model parameters. Rather than relying on purely numerical rewards, TPO translates reward signals into textual critiques and uses them as textual rewards to iteratively refine its response. Evaluations on benchmarks covering instruction following, preference alignment, safety, and mathematics reveal that TPO progressively improves alignment with human preferences. Notably, after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO scales efficiently with both the search width and depth during inference. Through case studies, we illustrate how TPO exploits the innate capacity of LLM to interpret and act upon reward signals. Our findings establish TPO as a practical, lightweight alternative for test-time preference optimization, achieving alignment on the fly. Our code is publicly available at https://github.com/yafuly/TPO.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 38 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

32 of 38 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  2. 2024Inference-Time Language Model Alignment via Integrated Value Guidance26 Sep
  3. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey4 Sep
  4. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment24 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  7. 2024sDPO: Don't Use Your Data All at Once28 Mar
  8. 2024ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization14 Feb
  9. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  10. 2023Align on the Fly: Adapting Chatbot Behavior to Established Norms26 Dec
  11. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  12. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
2 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
6 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  6. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 14.

41 papers.

Sort:

By area:

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  2. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  3. 2025Self-Supervised Prompt Optimizationcited by 37
  4. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 18
  5. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 15
  6. 2025R2ec: Towards Large Recommender Models with Reasoningcited by 13
  7. 2025Thinking Preference Optimizationcited by 8
  8. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
  9. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Futurecited by 5
  10. 2025EmoFeedback2: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedbackcited by 5
  11. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
  12. 2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
29 more
  1. 2025TTOM: Test-Time Optimization and Memorization for Compositional Video Generationcited by 4
  2. 2025W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Searchcited by 4
  3. 2025DEPO: Dual-Efficiency Preference Optimization for LLM Agentscited by 4
  4. 2025MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attackscited by 4
  5. 2026nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Spacecited by 4
  6. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 4
  7. 2025Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberationcited by 3
  8. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  9. 2025Self-Reflective Generation at Test Timecited by 3
  10. 2025TextBO: Bayesian Optimization in Language Space for Eval-Efficient Self-Improving AIcited by 3
  11. 2026When Do Tools and Planning Help Large Language Models Think? A Cost- and Latency-Aware Benchmarkcited by 3
  12. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAGcited by 3
  13. 2026Many-Shot CoT-ICL: Making In-Context Learning Truly Learncited by 3
  14. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Controlcited by 2
  15. 2025TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewardscited by 2
  16. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluationcited by 2
  17. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  18. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflectioncited by 2
  19. 2025MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimizationcited by 1
  20. 2026Autoregressive Direct Preference Optimizationcited by 1
  21. 2026Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptationcited by 1
  22. 2026Query-Conditioned Test-Time Self-Training for Large Language Modelscited by 1
  23. 2025Textual Self-attention Network: Test-Time Preference Optimization through Textual Gradient-based Attentioncited by 0
  24. 2026EEG-Based Brain-LLM Interface for Human Preference Aligned Generationcited by 0
  25. 2026Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failurescited by 0
  26. 2026EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptationcited by 0
  27. 2026Wait, am I Being Fair? Characterizing Deductive Stereotyping and Mitigating It with Fair-GCGcited by 0
  28. 2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learningcited by 0
  29. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026REAR: Test-time Preference Realignment through Reward Decomposition
  2. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  3. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  4. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
  5. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimation
  6. 2024Inference time LLM alignment in single and multidomain preference spectrum
  7. 2025On-the-fly Preference Alignment via Principle-Guided Decoding
  8. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following