# Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

arXiv: https://arxiv.org/abs/2108.11513 · HTML: https://mlsanity.com/paper/2108.11513/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 24 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.02107/) (2019)
- [AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations](https://mlsanity.com/paper/2002.11252/) (2020)
- [Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.11810/) (2019)
- [Neural Input Search for Large Scale Recommendation Models](https://mlsanity.com/paper/1907.04471/) (2019)
- [Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems](https://mlsanity.com/paper/2007.14523/) (2020)
- [Memory-efficient Embedding for Recommendations](https://mlsanity.com/paper/2006.14827/) (2020)
- [Graph Intention Network for Click-through Rate Prediction in Sponsored Search](https://mlsanity.com/paper/2103.16164/) (2021)

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Embedding in Recommender Systems: A Survey](https://mlsanity.com/paper/2310.18608/) (2023)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [Clustered Embedding Learning for Recommender Systems](https://mlsanity.com/paper/2302.01478/) (2023)
- [Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation](https://mlsanity.com/paper/2402.02855/) (2024)
- [A Comprehensive Survey on Automated Machine Learning for Recommendations](https://mlsanity.com/paper/2204.01390/) (2022)
- [CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems](https://mlsanity.com/paper/2506.19993/) (2025)
- [Proxy-based Item Representation for Attribute and Context-aware Recommendation](https://mlsanity.com/paper/2312.06145/) (2023)
- [On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective](https://mlsanity.com/paper/2411.13052/) (2024)
- [Efficient Multi-Task Learning via Generalist Recommender](https://mlsanity.com/paper/2504.05318/) (2025)
- [Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems](https://mlsanity.com/paper/2401.04408/) (2024)
- [DimGrow: Memory-Efficient Field-level Embedding Dimension Search](https://mlsanity.com/paper/2505.12683/) (2025)

## Abstract (arXiv)

Embedding learning for categorical features is crucial for the deep learning-based recommendation models (DLRMs). Each feature value is mapped to an embedding vector via an embedding learning process. Conventional methods configure a fixed and uniform embedding size to all feature values from the same feature field. However, such a configuration is not only sub-optimal for embedding learning but also memory costly. Existing methods that attempt to resolve these problems, either rule-based or neural architecture search (NAS)-based, need extensive efforts on the human design or network training. They are also not flexible in embedding size selection or in warm-start-based applications. In this paper, we propose a novel and effective embedding size selection scheme. Specifically, we design an Adaptively-Masked Twins-based Layer (AMTL) behind the standard embedding layer. AMTL generates a mask vector to mask the undesired dimensions for each embedding vector. The mask vector brings flexibility in selecting the dimensions and the proposed layer can be easily added to either untrained or trained DLRMs. Extensive experimental evaluations show that the proposed scheme outperforms competitive baselines on all the benchmark tasks, and is also memory-efficient, saving 60% memory usage without compromising any performance metrics.