FastE: Readout-Triggered Token Compression for LLM Embedding Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 23 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
13 of 23 references.
Closest ancestors · 1
Same direction, newest first
Other references · 12
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2022Token Merging: Your ViT But Fastercited by 995
- 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
- 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
- 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
- 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
- 2024Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?cited by 19
- 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
- 2026RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inferencecited by 4
- 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)cited by 2
- 2024Practical token pruning for foundation models in few-shot conversational virtual assistant systemscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
- 2023Just CHOP: Embarrassingly Simple LLM Compression
- 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
- 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
- 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
- 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
- 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
- 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple