# Guided Speculative Inference for Efficient Test-Time Alignment of LLMs

arXiv: https://arxiv.org/abs/2506.04118 · HTML: https://mlsanity.com/paper/2506.04118/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)
- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://mlsanity.com/paper/2205.10487/) (2022)
- [RL with KL penalties is better viewed as Bayesian inference](https://mlsanity.com/paper/2205.11275/) (2022)
- [Controlled Decoding from Language Models](https://mlsanity.com/paper/2310.17022/) (2023)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [Theoretical guarantees on the best-of-n alignment policy](https://mlsanity.com/paper/2401.01879/) (2024)
- [Reward-Guided Speculative Decoding for Efficient LLM Reasoning](https://mlsanity.com/paper/2501.19324/) (2025)
- [Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment](https://mlsanity.com/paper/2503.21878/) (2025)
- [TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling](https://mlsanity.com/paper/2410.16033/) (2024)
- [Speculative Streaming: Fast LLM Inference without Auxiliary Models](https://mlsanity.com/paper/2402.11131/) (2024)
- [Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism](https://mlsanity.com/paper/2406.03853/) (2024)
- [Block Verification Accelerates Speculative Decoding](https://mlsanity.com/paper/2403.10444/) (2024)
- [Soft Best-of-n Sampling for Model Alignment](https://mlsanity.com/paper/2505.03156/) (2025)

## Built on by (most cited first, 11 of 11)

- [Steering diffusion models with quadratic rewards: a fine-grained analysis](https://mlsanity.com/paper/2602.16570/) (2026)
- [Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling](https://mlsanity.com/paper/2510.05825/) (2025)
- [textttSPECS: Faster Test-Time Scaling through Speculative Drafts](https://mlsanity.com/paper/2506.15733/) (2025)
- [Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment](https://mlsanity.com/paper/2603.06797/) (2026)
- [Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference](https://mlsanity.com/paper/2603.07887/) (2026)
- [GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedback](https://mlsanity.com/paper/2603.27448/) (2026)
- [LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Models](https://mlsanity.com/paper/2601.10416/) (2026)
- [The Power of Test-Time Training for Approximate Sampling](https://mlsanity.com/paper/2606.11437/) (2026)
- [Selective Safety Steering via Value-Filtered Decoding](https://mlsanity.com/paper/2605.14746/) (2026)
- [ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment](https://mlsanity.com/paper/2609.21899/) (2026)
- [Universal Bounds for Self-Normalized Kullback-Leibler Estimation](https://mlsanity.com/paper/2610.12851/) (2026)

## Abstract (arXiv)

We propose Guided Speculative Inference (GSI), a novel algorithm for efficient reward-guided decoding in large language models. GSI combines soft best-of-n test-time scaling with a reward model r(x,y) and speculative samples from a small auxiliary model πS(ymid x). We provably approximate both the optimal tilted policy πβ,B(ymid x) propto πB(ymid x)exp(β,r(x,y)) of soft best-of-n under the base model πB, as well as the expected reward under the optimal policy. In experiments on reasoning benchmarks (MATH500, OlympiadBench, Minerva Math, MMLU-STEM, GSM8K) and across different model families, our method achieves higher accuracy than standard soft best-of-n with πS and reward-guided speculative decoding (Liao et al., 2025), and in certain settings even outperforms soft best-of-n with πB, while reducing end-to-end latency by up to 28%. The code is available at https://github.com/j-geuter/GSI .