mlsanityData to 14 Oct 2026

Guided Speculative Inference for Efficient Test-Time Alignment of LLMs

arXiv 2506.04118 ↗First posted 4 Jun 2025cs.LG stat.MLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

We propose Guided Speculative Inference (GSI), a novel algorithm for efficient reward-guided decoding in large language models. GSI combines soft best-of-n test-time scaling with a reward model r(x,y) and speculative samples from a small auxiliary model πS(ymid x). We provably approximate both the optimal tilted policy πβ,B(ymid x) propto πB(ymid x)exp(β,r(x,y)) of soft best-of-n under the base model πB, as well as the expected reward under the optimal policy. In experiments on reasoning benchmarks (MATH500, OlympiadBench, Minerva Math, MMLU-STEM, GSM8K) and across different model families, our method achieves higher accuracy than standard soft best-of-n with πS and reward-guided speculative decoding (Liao et al., 2025), and in certain settings even outperforms soft best-of-n with πB, while reducing end-to-end latency by up to 28%. The code is available at https://github.com/j-geuter/GSI .

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 41 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

32 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Soft Best-of-n Sampling for Model Alignment6 May
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment27 Mar
  3. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  4. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Models9 Jun
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2023Scaling Data-Constrained Language Modelscited by 403
15 more
  1. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  5. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  6. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  7. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  8. 2023Controlled Decoding from Language Modelscited by 135
  9. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  10. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  11. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  12. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 38
  13. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 32
  14. 2024Block Verification Accelerates Speculative Decodingcited by 30
  15. 2025Scaling Laws for Speculative Decodingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2026Steering diffusion models with quadratic rewards: a fine-grained analysiscited by 5
  2. 2025Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scalingcited by 4
  3. 2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
  4. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 3
  5. 2026Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inferencecited by 3
  6. 2026GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedbackcited by 3
  7. 2026LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Modelscited by 1
  8. 2026The Power of Test-Time Training for Approximate Samplingcited by 1
  9. 2026Selective Safety Steering via Value-Filtered Decodingcited by 0
  10. 2026ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignmentcited by 0
  11. 2026Universal Bounds for Self-Normalized Kullback-Leibler Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  2. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  3. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  5. 2025Speculative Decoding: Performance or Illusion?
  6. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
  7. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  8. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding