# Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data

arXiv: https://arxiv.org/abs/1807.00311 · HTML: https://mlsanity.com/paper/1807.00311/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2018.

## Built on by (most cited first, 30 of 44)

- [Deep Learning for Click-Through Rate Estimation](https://mlsanity.com/paper/2104.10584/) (2021)
- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction](https://mlsanity.com/paper/2310.09234/) (2023)
- [TabGNN: Multiplex Graph Neural Network for Tabular Data Prediction](https://mlsanity.com/paper/2108.09127/) (2021)
- [ARM-Net: Adaptive Relation Modeling Network for Structured Data](https://mlsanity.com/paper/2107.01830/) (2021)
- [GFS: Graph-based Feature Synthesis for Prediction over Relational Databases](https://mlsanity.com/paper/2312.02037/) (2023)
- [User Response Prediction in Online Advertising](https://mlsanity.com/paper/2101.02342/) (2021)
- [Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems](https://mlsanity.com/paper/2311.05884/) (2023)
- [ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table](https://mlsanity.com/paper/2104.08542/) (2021)
- [Memorize, Factorize, or be Na"ive: Learning Optimal Feature Interaction Methods for CTR Prediction](https://mlsanity.com/paper/2108.01265/) (2021)
- [MAP: A Model-agnostic Pretraining Framework for Click-through Rate Prediction](https://mlsanity.com/paper/2308.01737/) (2023)
- [VFLAIR: A Research Library and Benchmark for Vertical Federated Learning](https://mlsanity.com/paper/2310.09827/) (2023)
- [XCrossNet: Feature Structure-Oriented Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2104.10907/) (2021)
- [Learning Enhanced Representations for Tabular Data via Neighborhood Propagation](https://mlsanity.com/paper/2206.06587/) (2022)
- [Muddling Label Regularization: Deep Learning for Tabular Datasets](https://mlsanity.com/paper/2106.04462/) (2021)
- [i-Razor: A Differentiable Neural Input Razor for Feature Selection and Dimension Search in DNN-Based Recommender Systems](https://mlsanity.com/paper/2204.00281/) (2022)
- [A Comprehensive Survey on Automated Machine Learning for Recommendations](https://mlsanity.com/paper/2204.01390/) (2022)
- [From Principle to Practice: Vertical Data Minimization for Machine Learning](https://mlsanity.com/paper/2311.10500/) (2023)
- [Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction](https://mlsanity.com/paper/2505.17999/) (2025)
- [Task-wise Split Gradient Boosting Trees for Multi-center Diabetes Prediction](https://mlsanity.com/paper/2108.07107/) (2021)
- [Secure Your Ride: Real-time Matching Success Rate Prediction for Passenger-Driver Pairs](https://mlsanity.com/paper/2109.07571/) (2021)
- [AEFE: Automatic Embedded Feature Engineering for Categorical Features](https://mlsanity.com/paper/2110.09770/) (2021)
- [On-Device Model Fine-Tuning with Label Correction in Recommender Systems](https://mlsanity.com/paper/2211.01163/) (2022)
- [xDeepInt: a hybrid architecture for modeling the vector-wise and bit-wise feature interactions](https://mlsanity.com/paper/2301.01089/) (2023)
- [Exploiting Field Dependencies for Learning on Categorical Data](https://mlsanity.com/paper/2307.09321/) (2023)
- [Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network](https://mlsanity.com/paper/2310.15342/) (2023)
- [DisCo: Towards Harmonious Disentanglement and Collaboration between Tabular and Semantic Space for Recommendation](https://mlsanity.com/paper/2406.00011/) (2024)
- [MOEF: Modeling Occasion Evolution in Frequency Domain for Promotion-Aware Click-Through Rate Prediction](https://mlsanity.com/paper/2112.13747/) (2021)
- [Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation](https://mlsanity.com/paper/2211.11159/) (2022)
- [AdaEnsemble: Learning Adaptively Sparse Structured Ensemble Network for Click-Through Rate Prediction](https://mlsanity.com/paper/2301.08353/) (2023)

## Abstract (arXiv)

User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.