# SHORING: Design Provable Conditional High-Order Interaction Network via Symbolic Testing

arXiv: https://arxiv.org/abs/2107.01326 · HTML: https://mlsanity.com/paper/2107.01326/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 3 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Adversarial Attack on Graph Structured Data](https://mlsanity.com/paper/1806.02371/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Neural Factorization Machines for Sparse Predictive Analytics](https://mlsanity.com/paper/1708.05027/) (2017)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://mlsanity.com/paper/1708.04617/) (2017)
- [Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09248/) (2019)
- [TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning](https://mlsanity.com/paper/2105.07944/) (2021)
- [Deep Learning over Multi-field Categorical Data: A Case Study on User Response Prediction](https://mlsanity.com/paper/1601.02376/) (2016)
- [Generative Adversarial User Model for Reinforcement Learning Based Recommendation System](https://mlsanity.com/paper/1812.10613/) (2018)
- [Deep Session Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.06482/) (2019)
- [Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions](https://mlsanity.com/paper/1909.03276/) (2019)
- [Higher-Order Factorization Machines](https://mlsanity.com/paper/1607.07195/) (2016)
- [Double Neural Counterfactual Regret Minimization](https://mlsanity.com/paper/1812.10607/) (2018)
- [Exponential Family Embeddings](https://mlsanity.com/paper/1608.00778/) (2016)
- [Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms](https://mlsanity.com/paper/1607.08810/) (2016)
- [Automatic trajectory measurement of large numbers of crowded objects](https://mlsanity.com/paper/1902.00835/) (2019)

## Abstract (arXiv)

Deep learning provides a promising way to extract effective representations from raw data in an end-to-end fashion and has proven its effectiveness in various domains such as computer vision, natural language processing, etc. However, in domains such as content/product recommendation and risk management, where sequence of event data is the most used raw data form and experts derived features are more commonly used, deep learning models struggle to dominate the game. In this paper, we propose a symbolic testing framework that helps to answer the question of what kinds of expert-derived features could be learned by a neural network. Inspired by this testing framework, we introduce an efficient architecture named SHORING, which contains two components: textitevent network and textitsequence network. The textitevent network learns arbitrarily yet efficiently high-order textitevent-level embeddings via a provable reparameterization trick, the textitsequence network aggregates from sequence of textitevent-level embeddings. We argue that SHORING is capable of learning certain standard symbolic expressions which the standard multi-head self-attention network fails to learn, and conduct comprehensive experiments and ablation studies on four synthetic datasets and three real-world datasets. The results show that SHORING empirically outperforms the state-of-the-art methods.