# Memorize, Factorize, or be Na"ive: Learning Optimal Feature Interaction Methods for CTR Prediction

arXiv: https://arxiv.org/abs/2108.01265 · HTML: https://mlsanity.com/paper/2108.01265/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 25 of 25)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Neural Architecture Optimization](https://mlsanity.com/paper/1808.07233/) (2018)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Deep Learning over Multi-field Categorical Data: A Case Study on User Response Prediction](https://mlsanity.com/paper/1601.02376/) (2016)
- [Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://mlsanity.com/paper/1806.03514/) (2018)
- [AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction](https://mlsanity.com/paper/2003.11235/) (2020)
- [AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations](https://mlsanity.com/paper/2002.11252/) (2020)
- [Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data](https://mlsanity.com/paper/1807.00311/) (2018)
- [An Embedding Learning Framework for Numerical Features in CTR Prediction](https://mlsanity.com/paper/2012.08986/) (2020)
- [FM2: Field-matrixed Factorization Machines for Recommender Systems](https://mlsanity.com/paper/2102.12994/) (2021)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)
- [Detecting Beneficial Feature Interactions for Recommender Systems](https://mlsanity.com/paper/2008.00404/) (2020)
- [Directional Pruning of Deep Neural Networks](https://mlsanity.com/paper/2006.09358/) (2020)
- [Memory-efficient Embedding for Recommendations](https://mlsanity.com/paper/2006.14827/) (2020)
- [Neural Graph Matching based Collaborative Filtering](https://mlsanity.com/paper/2105.04067/) (2021)
- [AutoFT: Automatic Fine-Tune for Parameters Transfer Learning in Click-Through Rate Prediction](https://mlsanity.com/paper/2106.04873/) (2021)
- [A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction](https://mlsanity.com/paper/2105.10484/) (2021)

## Built on by (most cited first, 12 of 12)

- [Gradient-based Bi-level Optimization for Deep Learning: A Survey](https://mlsanity.com/paper/2207.11719/) (2022)
- [Explicit Feature Interaction-aware Uplift Network for Online Marketing](https://mlsanity.com/paper/2306.00315/) (2023)
- [MvFS: Multi-view Feature Selection for Recommender System](https://mlsanity.com/paper/2309.02064/) (2023)
- [OptDist: Learning Optimal Distribution for Customer Lifetime Value Prediction](https://mlsanity.com/paper/2408.08585/) (2024)
- [A Comprehensive Survey on Automated Machine Learning for Recommendations](https://mlsanity.com/paper/2204.01390/) (2022)
- [CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU](https://mlsanity.com/paper/2204.06240/) (2022)
- [Fusion Matters: Learning Fusion in Deep Click-through Rate Prediction Models](https://mlsanity.com/paper/2411.15731/) (2024)
- [RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems](https://mlsanity.com/paper/2607.29241/) (2026)
- [Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network](https://mlsanity.com/paper/2310.15342/) (2023)
- [Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation](https://mlsanity.com/paper/2312.15112/) (2023)
- [All-in-One: Heterogeneous Interaction Modeling for Cold-Start Rating Prediction](https://mlsanity.com/paper/2403.17740/) (2024)
- [Curriculum Modeling the Dependence among Targets with Multi-task Learning for Financial Marketing](https://mlsanity.com/paper/2305.01514/) (2023)

## Abstract (arXiv)

Click-through rate prediction is one of the core tasks in commercial recommender systems. It aims to predict the probability of a user clicking a particular item given user and item features. As feature interactions bring in non-linearity, they are widely adopted to improve the performance of CTR prediction models. Therefore, effectively modelling feature interactions has attracted much attention in both the research and industry field. The current approaches can generally be categorized into three classes: (1) na"ive methods, which do not model feature interactions and only use original features; (2) memorized methods, which memorize feature interactions by explicitly viewing them as new features and assigning trainable embeddings; (3) factorized methods, which learn latent vectors for original features and implicitly model feature interactions through factorization functions. Studies have shown that modelling feature interactions by one of these methods alone are suboptimal due to the unique characteristics of different feature interactions. To address this issue, we first propose a general framework called OptInter which finds the most suitable modelling method for each feature interaction. Different state-of-the-art deep CTR models can be viewed as instances of OptInter. To realize the functionality of OptInter, we also introduce a learning algorithm that automatically searches for the optimal modelling method. We conduct extensive experiments on four large datasets. Our experiments show that OptInter improves the best performed state-of-the-art baseline deep CTR models by up to 2.21%. Compared to the memorized method, which also outperforms baselines, we reduce up to 91% parameters. In addition, we conduct several ablation studies to investigate the influence of different components of OptInter. Finally, we provide interpretable discussions on the results of OptInter.