# Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation

arXiv: https://arxiv.org/abs/2103.03578 · HTML: https://mlsanity.com/paper/2103.03578/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2021.

## Built on by (most cited first, 30 of 39)

- [Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender System](https://mlsanity.com/paper/2404.11343/) (2024)
- [Multimodal Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03883/) (2023)
- [Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2408.09698/) (2024)
- [Decoupled Side Information Fusion for Sequential Recommendation](https://mlsanity.com/paper/2204.11046/) (2022)
- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [A Survey on User Behavior Modeling in Recommender Systems](https://mlsanity.com/paper/2302.11087/) (2023)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2501.14269/) (2025)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Text Matching Improves Sequential Recommendation by Reducing Popularity Biases](https://mlsanity.com/paper/2308.14029/) (2023)
- [Relation-aware Heterogeneous Graph for User Profiling](https://mlsanity.com/paper/2110.07181/) (2021)
- [Temporal Interest Network for User Response Prediction](https://mlsanity.com/paper/2308.08487/) (2023)
- [Robust Image Semantic Coding with Learnable CSI Fusion Masking over MIMO Fading Channels](https://mlsanity.com/paper/2406.07389/) (2024)
- [Coarse-to-Fine Sparse Sequential Recommendation](https://mlsanity.com/paper/2204.01839/) (2022)
- [PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregation](https://mlsanity.com/paper/2305.12349/) (2023)
- [Dual Contrastive Transformer for Hierarchical Preference Modeling in Sequential Recommendation](https://mlsanity.com/paper/2410.22790/) (2024)
- [A Survey on Large Language Models in Multimodal Recommender Systems](https://mlsanity.com/paper/2505.09777/) (2025)
- [Preference Discerning with LLM-Enhanced Generative Retrieval](https://mlsanity.com/paper/2412.08604/) (2024)
- [Intention Knowledge Graph Construction for User Intention Relation Modeling](https://mlsanity.com/paper/2412.11500/) (2024)
- [Diffusion-based Multi-modal Synergy Interest Network for Click-through Rate Prediction](https://mlsanity.com/paper/2508.21460/) (2025)
- [SanMove: Next Location Recommendation via Self-Attention Network](https://mlsanity.com/paper/2112.09076/) (2021)
- [Deep Stable Multi-Interest Learning for Out-of-distribution Sequential Recommendation](https://mlsanity.com/paper/2304.05615/) (2023)
- [Improving Sequential Recommendation Models with an Enhanced Loss Function](https://mlsanity.com/paper/2301.00979/) (2023)
- [Learning to Personalize Recommendation based on Customers' Shopping Intents](https://mlsanity.com/paper/2305.05279/) (2023)
- [BiVRec: Bidirectional View-based Multimodal Sequential Recommendation](https://mlsanity.com/paper/2402.17334/) (2024)
- [Attention-based sequential recommendation system using multimodal data](https://mlsanity.com/paper/2405.17959/) (2024)
- [GCN-TULHOR: Trajectory-User Linking Leveraging GCNs and Higher-Order Spatial Representations](https://mlsanity.com/paper/2509.11095/) (2025)
- [HoMer: Addressing Heterogeneities by Modeling Sequential and Set-wise Contexts for CTR Prediction](https://mlsanity.com/paper/2510.11100/) (2025)

## Abstract (arXiv)

Sequential recommender systems aim to model users' evolving interests from their historical behaviors, and hence make customized time-relevant recommendations. Compared with traditional models, deep learning approaches such as CNN and RNN have achieved remarkable advancements in recommendation tasks. Recently, the BERT framework also emerges as a promising method, benefited from its self-attention mechanism in processing sequential data. However, one limitation of the original BERT framework is that it only considers one input source of the natural language tokens. It is still an open question to leverage various types of information under the BERT framework. Nonetheless, it is intuitively appealing to utilize other side information, such as item category or tag, for more comprehensive depictions and better recommendations. In our pilot experiments, we found naive approaches, which directly fuse types of side information into the item embeddings, usually bring very little or even negative effects. Therefore, in this paper, we propose the NOninVasive self-attention mechanism (NOVA) to leverage side information effectively under the BERT framework. NOVA makes use of side information to generate better attention distribution, rather than directly altering the item embedding, which may cause information overwhelming. We validate the NOVA-BERT model on both public and commercial datasets, and our method can stably outperform the state-of-the-art models with negligible computational overheads.