MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems. However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, cross-encoder ranking systems often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information. To this end, we propose MixLM, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of cross-encoder rankers. In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use mix-interaction, a mixture of text and embedding tokens to represent the input. Specifically, MixLM encodes all items in the catalog into a few embedding tokens and stores in a nearline cache. The encoded item descriptions are used during online inference, effectively reducing the item length from a few thousand text tokens to a few embedding tokens. We share insights from deploying our MixLM framework to a real-world search application at LinkedIn, including a detailed discussion of our training pipelines, as well as a thorough analysis of our online serving infrastructure optimization. With the same latency budget and on-par relevance metrics, MixLM increased throughput by 10.0x comparing with strong baselines, 75.9x over full-text LLM rerankers. The efficiency gains delivered by MixLM enabled full-traffic deployment of LLM-powered search, which resulted in a significant 0.47% increase in Daily Active Users (DAU) in online A/B tests.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 25 Nov 2025.
Builds on
30 of 46 references.
References · 30
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
- 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
- 2023A Survey on Large Language Models for Recommendationcited by 379
- 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
- 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
18 more
- 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
- 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
- 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 165
- 2020Embedding-based Retrieval in Facebook Searchcited by 156
- 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
- 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
- 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
- 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 77
- 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
- 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 55
- 2023A Two-Stage Adaptation of Large Language Models for Text Rankingcited by 18
- 2020Understanding BERT Rankers Under Distillationcited by 15
- 2023Could Small Language Models Serve as Recommenders? Towards Data-centric Cold-start Recommendationscited by 15
- 2024SyNeg: LLM-Driven Synthetic Hard-Negatives for Dense Retrievalcited by 14
- 2023RD-Suite: A Benchmark for Ranking Distillationcited by 8
- 2025From Reasoning LLMs to BERT: A Two-Stage Distillation Framework for Search Relevancecited by 7
- 2025Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Modelscited by 4
- 2025Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.