# Beyond Two-Tower Matching: Learning Sparse Retrievable Cross-Interactions for Recommendation

arXiv: https://arxiv.org/abs/2311.18213 · HTML: https://mlsanity.com/paper/2311.18213/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Embedding-based Retrieval in Facebook Search](https://mlsanity.com/paper/2006.11632/) (2020)
- [Neural Collaborative Filtering vs. Matrix Factorization Revisited](https://mlsanity.com/paper/2005.09683/) (2020)
- [MeLU: Meta-Learned User Preference Estimator for Cold-Start Recommendation](https://mlsanity.com/paper/1908.00413/) (2019)
- [Multi-Interest Network with Dynamic Routing for Recommendation at Tmall](https://mlsanity.com/paper/1904.08030/) (2019)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09433/) (2019)
- [SimpleX: A Simple and Strong Baseline for Collaborative Filtering](https://mlsanity.com/paper/2109.12613/) (2021)
- [BARS: Towards Open Benchmarking for Recommender Systems](https://mlsanity.com/paper/2205.09626/) (2022)
- [Open Benchmarking for Click-Through Rate Prediction](https://mlsanity.com/paper/2009.05794/) (2020)
- [Cross-Modal Discrete Representation Learning](https://mlsanity.com/paper/2106.05438/) (2021)
- [A Semi-Personalized System for User Cold Start Recommendation on Music Streaming Apps](https://mlsanity.com/paper/2106.03819/) (2021)
- [Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations](https://mlsanity.com/paper/2109.13059/) (2021)
- [PEAR: Personalized Re-ranking with Contextualized Transformer for Recommendation](https://mlsanity.com/paper/2203.12267/) (2022)
- [Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index](https://mlsanity.com/paper/2105.03933/) (2021)

## Built on by (most cited first, 13 of 13)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [ERASE: Error-Resilient Representation Learning on Graphs for Label Noise Tolerance](https://mlsanity.com/paper/2312.08852/) (2023)
- [CF-KAN: Kolmogorov-Arnold Network-based Collaborative Filtering to Mitigate Catastrophic Forgetting in Recommender Systems](https://mlsanity.com/paper/2409.05878/) (2024)
- [Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications](https://mlsanity.com/paper/2510.21131/) (2025)
- [Unleashing the Potential of Two-Tower Models: Diffusion-Based Cross-Interaction for Large-Scale Matching](https://mlsanity.com/paper/2502.20687/) (2025)
- [InteractRank: Personalized Web-Scale Search Pre-Ranking with Cross Interaction Features](https://mlsanity.com/paper/2504.06609/) (2025)
- [MTMD: A Multi-Task Multi-Domain Framework for Unified Ad Lightweight Ranking at Pinterest](https://mlsanity.com/paper/2510.09857/) (2025)
- [Target-Aware Early Stage Ranking](https://mlsanity.com/paper/2511.21095/) (2025)
- [Towards a Theoretical Understanding of Two Tower Recommendation Models](https://mlsanity.com/paper/2403.00802/) (2024)
- [Capturing Fine-Grained Alignments Improves 3D Affordance Detection](https://mlsanity.com/paper/2506.19312/) (2025)
- [A Simple and Effective Framework for Symmetric Consistent Indexing in Large-Scale Dense Retrieval](https://mlsanity.com/paper/2512.13074/) (2025)
- [Keeping the Index Open: The Recommendation-Side Cost of Shared Search and Recommendation](https://mlsanity.com/paper/2608.24079/) (2026)
- [Connected Content Retriever: Dense Graph Edge Features Powering Pre-Ranking at LinkedIn](https://mlsanity.com/paper/2609.22441/) (2026)

## Abstract (arXiv)

Two-tower models are a prevalent matching framework for recommendation, which have been widely deployed in industrial applications. The success of two-tower matching attributes to its efficiency in retrieval among a large number of items, since the item tower can be precomputed and used for fast Approximate Nearest Neighbor (ANN) search. However, it suffers two main challenges, including limited feature interaction capability and reduced accuracy in online serving. Existing approaches attempt to design novel late interactions instead of dot products, but they still fail to support complex feature interactions or lose retrieval efficiency. To address these challenges, we propose a new matching paradigm named SparCode, which supports not only sophisticated feature interactions but also efficient retrieval. Specifically, SparCode introduces an all-to-all interaction module to model fine-grained query-item interactions. Besides, we design a discrete code-based sparse inverted index jointly trained with the model to achieve effective and efficient model inference. Extensive experiments have been conducted on open benchmark datasets to demonstrate the superiority of our framework. The results show that SparCode significantly improves the accuracy of candidate item matching while retaining the same level of retrieval efficiency with two-tower models. Our source code will be available at MindSpore/models.