# FlashEvaluator: Expanding Search Space with Parallel Sequence-Level Evaluation

arXiv: https://arxiv.org/abs/2603.02565 · HTML: https://mlsanity.com/paper/2603.02565/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 77 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [A vector-contraction inequality for Rademacher complexities](https://mlsanity.com/paper/1605.00251/) (2016)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Learning a Deep Listwise Context Model for Ranking Refinement](https://mlsanity.com/paper/1804.05936/) (2018)
- [Quality-Aware Decoding for Neural Machine Translation](https://mlsanity.com/paper/2205.00978/) (2022)
- [SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval](https://mlsanity.com/paper/1912.05891/) (2019)
- [Generative Flow Network for Listwise Recommendation](https://mlsanity.com/paper/2306.02239/) (2023)
- [Non-autoregressive Generative Models for Reranking Recommendation](https://mlsanity.com/paper/2402.06871/) (2024)
- [Revisit Recommender System in the Permutation Prospective](https://mlsanity.com/paper/2102.12057/) (2021)
- [GRN: Generative Rerank Network for Context-wise Recommendation](https://mlsanity.com/paper/2104.00860/) (2021)
- [PEAR: Personalized Re-ranking with Contextualized Transformer for Recommendation](https://mlsanity.com/paper/2203.12267/) (2022)
- [Variation Control and Evaluation for Generative SlateRecommendations](https://mlsanity.com/paper/2102.13302/) (2021)
- [Multi-Level Interaction Reranking with User Behavior History](https://mlsanity.com/paper/2204.09370/) (2022)

## Abstract (arXiv)

The Generator-Evaluator (G-E) framework generates K candidate sequences and uses an evaluator to select the highest-scoring one, which is widely used in recommender systems (RecSys) and natural language processing (NLP). Existing evaluators commonly score candidates independently. Although such evaluations can be batched, independent scoring neither models interactions among candidates nor eliminates repeated computation of request-level context and recurring candidate elements, causing the total evaluation work to grow approximately linearly with K. To handle with, we propose FlashEvaluator, a joint evaluator that scores all candidate sequences in a single forward pass. FlashEvaluator factorizes evaluation into shared request-level encoding, reusable candidate-side computation, sequence assembly by indexing, and cross-sequence interaction for setwise comparison. We call this request-local reuse scheme QKV-Cache: inspired by autoregressive KV caching, it reuses context-side key/value representations across candidate sequences and, when candidate elements recur, reuses their request-conditioned representations on the query side. In repeated-item settings, the dominant item-encoding cost therefore depends on the number of distinct items rather than their total occurrences across sequences, reducing the marginal cost of evaluating additional candidates. We provide a computational analysis and evaluate FlashEvaluator on recommendation and text summarization. The results show lower latency and higher throughput with competitive recommendation and summarization quality. In an online deployment at Kuaishou with K=50, FlashEvaluator reduces inference latency by 44% and increases QPS by 114% relative to the production baseline, while yielding statistically significant gains in retention, engagement, and ecosystem metrics.