# Inferring from Logits: Exploring Best Practices for Decoding-Free Generative Candidate Selection

arXiv: https://arxiv.org/abs/2501.17338 · HTML: https://mlsanity.com/paper/2501.17338/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Builds on (most cited first, 22 of 22)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Dense Text Retrieval based on Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.14876/) (2022)
- [Code4Struct: Code Generation for Few-Shot Event Structure Prediction](https://mlsanity.com/paper/2210.12810/) (2022)
- [Learning to Tokenize for Generative Retrieval](https://mlsanity.com/paper/2304.04171/) (2023)
- [DICE: Data-Efficient Clinical Event Extraction with Generative Models](https://mlsanity.com/paper/2208.07989/) (2022)
- [HyperExpan: Taxonomy Expansion with Hyperbolic Representation Learning](https://mlsanity.com/paper/2109.10500/) (2021)
- [Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?](https://mlsanity.com/paper/2212.10784/) (2022)
- [Multi-hop Evidence Retrieval for Cross-document Relation Extraction](https://mlsanity.com/paper/2212.10786/) (2022)
- [Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning](https://mlsanity.com/paper/2301.10915/) (2023)
- [Mitigating Bias for Question Answering Models by Tracking Bias Influence](https://mlsanity.com/paper/2310.08795/) (2023)

## Built on by (most cited first, 1 of 1)

- [DDO: Dual-Decision Optimization for LLM-Based Medical Consultation via Multi-Agent Collaboration](https://mlsanity.com/paper/2505.18630/) (2025)

## Abstract (arXiv)

Generative Language Models rely on autoregressive decoding to produce the output sequence token by token. Many tasks such as preference optimization, require the model to produce task-level output consisting of multiple tokens directly by selecting candidates from a pool as predictions. Determining a task-level prediction from candidates using the ordinary token-level decoding mechanism is constrained by time-consuming decoding and interrupted gradients by discrete token selection. Existing works have been using decoding-free candidate selection methods to obtain candidate probability from initial output logits over vocabulary. Though these estimation methods are widely used, they are not systematically evaluated, especially on end tasks. We introduce an evaluation of a comprehensive collection of decoding-free candidate selection approaches on a comprehensive set of tasks, including five multiple-choice QA tasks with a small candidate pool and four clinical decision tasks with a massive amount of candidates, some with 10k+ options. We evaluate the estimation methods paired with a wide spectrum of foundation LMs covering different architectures, sizes and training paradigms. The results and insights from our analysis inform the future model design.