mlsanityData to 14 Oct 2026

Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

arXiv 2508.05165 ↗First posted 7 Aug 2025cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning LLMs with user preferences is crucial for real-world use but often requires costly fine-tuning or expensive inference, forcing trade-offs between alignment quality and computational cost. Existing inference-time methods typically ignore this balance, focusing solely on the optimized policy's performance. We propose HIA (Heuristic-Guided Inference-time Alignment), a tuning-free, black-box-compatible approach that uses a lightweight prompt optimizer, heuristic reward models, and two-stage filtering to reduce inference calls while preserving alignment quality. On real-world prompt datasets, HelpSteer and ComPRed, HIA outperforms best-of-N sampling, beam search, and greedy search baselines in multi-objective, goal-conditioned tasks under the same inference budget. We also find that HIA is effective under low-inference budgets with as little as one or two response queries, offering a practical solution for scalable, personalized LLM deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 42 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

30 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models13 Nov
  2. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  4. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
  5. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  11. 2024A Survey on Large Language Models for Code Generationcited by 658
  12. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
12 more
  1. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  2. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  4. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  5. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  6. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  7. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  8. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  9. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  10. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  11. 2024ComPO: Community Preferences for Language Model Personalizationcited by 19
  12. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
  2. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
  3. 2024Inference time LLM alignment in single and multidomain preference spectrum
  4. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  5. 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization
  6. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  7. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  8. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective