# On the Rejection Criterion for Proxy-based Test-time Alignment

arXiv: https://arxiv.org/abs/2604.16146 · HTML: https://mlsanity.com/paper/2604.16146/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model](https://mlsanity.com/paper/2310.09520/) (2023)
- [Nudging: Inference-time Alignment of LLMs via Guided Decoding](https://mlsanity.com/paper/2410.09300/) (2024)
- [Sample, Don't Search: Rethinking Test-Time Alignment for Language Models](https://mlsanity.com/paper/2504.03790/) (2025)
- [Systematic Rectification of Language Models via Dead-end Analysis](https://mlsanity.com/paper/2302.14003/) (2023)
- [A Critical Look At Tokenwise Reward-Guided Text Generation](https://mlsanity.com/paper/2406.07780/) (2024)
- [Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability](https://mlsanity.com/paper/2502.11115/) (2025)
- [Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner](https://mlsanity.com/paper/2508.15044/) (2025)
- [On the Low-Rank Parametrization of Reward Models for Controlled Language Generation](https://mlsanity.com/paper/2407.04615/) (2024)
- [Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral](https://mlsanity.com/paper/2510.27017/) (2025)

## Abstract (arXiv)

Recent works proposed test-time alignment methods that rely on a small aligned model as a proxy that guides the generation of a larger base (unaligned) model. The implicit reward approach skews the large model distribution, whereas the nudging approach defers the generation of the next token to the small aligned model when the large base one is unconfident about its outcome. In this work, we first show that both approaches can be reduced to sampling from similar graphical models, where they differ only in the definition of a rejection criterion (or distribution). Moreover, we argue that the confidence criterion is ill-motivated due to linguistic phenomena like ambiguous phrasing. We propose a novel rejection criterion based on a conservative confidence bet. Experimentally, our novel approach outperforms previous work on several datasets.