# Fast Variational AutoEncoder with Inverted Multi-Index for Collaborative Filtering

arXiv: https://arxiv.org/abs/2109.05773 · HTML: https://mlsanity.com/paper/2109.05773/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [Sequential Recommender Systems: Challenges, Progress and Prospects](https://mlsanity.com/paper/2001.04830/) (2019)
- [Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)](https://mlsanity.com/paper/1405.5869/) (2014)
- [On Using Very Large Target Vocabulary for Neural Machine Translation](https://mlsanity.com/paper/1412.2007/) (2014)
- [IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models](https://mlsanity.com/paper/1705.10513/) (2017)
- [RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback](https://mlsanity.com/paper/1912.11160/) (2019)
- [Adaptive Sampled Softmax with Kernel Based Sampling](https://mlsanity.com/paper/1712.00527/) (2017)
- [Security and Privacy on Blockchain](https://mlsanity.com/paper/1903.07602/) (2019)
- [Sequential Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1811.09975/) (2018)
- [Quantization based Fast Inner Product Search](https://mlsanity.com/paper/1509.01469/) (2015)
- [An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family](https://mlsanity.com/paper/1511.05042/) (2015)
- [On Symmetric and Asymmetric LSHs for Inner Product Search](https://mlsanity.com/paper/1410.5518/) (2014)
- [On distinguishability criteria for estimating generative models](https://mlsanity.com/paper/1412.6515/) (2014)
- [A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models](https://mlsanity.com/paper/1703.05160/) (2017)
- [Sampling-Decomposable Generative Adversarial Recommender](https://mlsanity.com/paper/2011.00956/) (2020)
- [Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner Product Search (MIPS)](https://mlsanity.com/paper/1410.5410/) (2014)
- [Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets](https://mlsanity.com/paper/1412.7091/) (2014)
- [LightRNN: Memory and Computation-Efficient Recurrent Neural Networks](https://mlsanity.com/paper/1610.09893/) (2016)
- [Correlated Variational Auto-Encoders](https://mlsanity.com/paper/1905.05335/) (2019)

## Built on by (most cited first, 8 of 8)

- [CONVERT:Contrastive Graph Clustering with Reliable Augmentation](https://mlsanity.com/paper/2308.08963/) (2023)
- [Mixed Graph Contrastive Network for Semi-Supervised Node Classification](https://mlsanity.com/paper/2206.02796/) (2022)
- [Dual Test-time Training for Out-of-distribution Recommender System](https://mlsanity.com/paper/2407.15620/) (2024)
- [BR-SNIS: Bias Reduced Self-Normalized Importance Sampling](https://mlsanity.com/paper/2207.06364/) (2022)
- [Cooperative Retriever and Ranker in Deep Recommenders](https://mlsanity.com/paper/2206.14649/) (2022)
- [A Comprehensive Survey on Self-Supervised Learning for Recommendation](https://mlsanity.com/paper/2404.03354/) (2024)
- [Deep Task-specific Bottom Representation Network for Multi-Task Recommendation](https://mlsanity.com/paper/2308.05996/) (2023)
- [A Survey on Deep Neural Networks in Collaborative Filtering Recommendation Systems](https://mlsanity.com/paper/2412.01378/) (2024)

## Abstract (arXiv)

Variational AutoEncoder (VAE) has been extended as a representative nonlinear method for collaborative filtering. However, the bottleneck of VAE lies in the softmax computation over all items, such that it takes linear costs in the number of items to compute the loss and gradient for optimization. This hinders the practical use due to millions of items in real-world scenarios. Importance sampling is an effective approximation method, based on which the sampled softmax has been derived. However, existing methods usually exploit the uniform or popularity sampler as proposal distributions, leading to a large bias of gradient estimation. To this end, we propose to decompose the inner-product-based softmax probability based on the inverted multi-index, leading to sublinear-time and highly accurate sampling. Based on the proposed proposals, we develop a fast Variational AutoEncoder (FastVAE) for collaborative filtering. FastVAE can outperform the state-of-the-art baselines in terms of both sampling quality and efficiency according to the experiments on three real-world datasets.