# A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation

arXiv: https://arxiv.org/abs/2207.07483 · HTML: https://mlsanity.com/paper/2207.07483/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 74 references that are in the map.
- First posted on arXiv 15 Jul 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms](https://mlsanity.com/paper/2011.01731/) (2020)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [A Simple Convolutional Generative Network for Next Item Recommendation](https://mlsanity.com/paper/1808.05163/) (2018)
- [Hierarchical Gating Networks for Sequential Recommendation](https://mlsanity.com/paper/1906.09217/) (2019)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer](https://mlsanity.com/paper/2108.06625/) (2021)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer](https://mlsanity.com/paper/2105.00522/) (2021)
- [Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation](https://mlsanity.com/paper/2103.03578/) (2021)
- [Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2005.12964/) (2020)
- [Black-Box Attacks on Sequential Recommenders via Data-Free Model Extraction](https://mlsanity.com/paper/2109.01165/) (2021)
- [On Model Stability as a Function of Random Seed](https://mlsanity.com/paper/1909.10447/) (2019)
- [Metadata Embeddings for User and Item Cold-start Recommendations](https://mlsanity.com/paper/1507.08439/) (2015)
- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [Lightweight Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/2108.11333/) (2021)

## Built on by (most cited first, 22 of 22)

- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling](https://mlsanity.com/paper/2308.07192/) (2023)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendation](https://mlsanity.com/paper/2306.14462/) (2023)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Scaling Sequential Recommendation Models with Transformers](https://mlsanity.com/paper/2412.07585/) (2024)
- [Autoregressive Generation Strategies for Top-K Sequential Recommendations](https://mlsanity.com/paper/2409.17730/) (2024)
- [Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Models](https://mlsanity.com/paper/2410.17276/) (2024)
- [Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study](https://mlsanity.com/paper/2301.05174/) (2023)
- [eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion](https://mlsanity.com/paper/2508.06450/) (2025)
- [Proxy-based Item Representation for Attribute and Context-aware Recommendation](https://mlsanity.com/paper/2312.06145/) (2023)
- [DenseRec: Revisiting Dense Content Embeddings for Sequential Transformer-based Recommendation](https://mlsanity.com/paper/2508.18442/) (2025)
- [Improving Sequential Recommendation Models with an Enhanced Loss Function](https://mlsanity.com/paper/2301.00979/) (2023)
- [Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement Learning](https://mlsanity.com/paper/2403.04875/) (2024)
- [ImplicitSLIM and How it Improves Embedding-based Collaborative Filtering](https://mlsanity.com/paper/2406.00198/) (2024)
- [Modeling and Analyzing the Influence of Non-Item Pages on Sequential Next-Item Prediction](https://mlsanity.com/paper/2408.15953/) (2024)
- [Compositions of Variant Experts for Integrating Short-Term and Long-Term Preferences](https://mlsanity.com/paper/2506.23170/) (2025)
- [Multi-Item-Query Attention for Stable Sequential Recommendation](https://mlsanity.com/paper/2509.24424/) (2025)
- [Sensory-Aware Sequential Recommendation via Review-Distilled Representations](https://mlsanity.com/paper/2603.02709/) (2026)
- [Retrieve-then-Adapt: Retrieval-Augmented Test-Time Adaptation for Sequential Recommendation](https://mlsanity.com/paper/2604.05379/) (2026)
- [SISA-Rec: A Semantically Integrated Sequential Recommender with Contrastive Alignment](https://mlsanity.com/paper/2607.11168/) (2026)
- [Balancing Trial and Reorder: A Hybrid Sequential Transformer-GBDT Ranker for On-Demand Delivery](https://mlsanity.com/paper/2609.16407/) (2026)

## Abstract (arXiv)

BERT4Rec is an effective model for sequential recommendation based on the Transformer architecture. In the original publication, BERT4Rec claimed superiority over other available sequential recommendation approaches (e.g. SASRec), and it is now frequently being used as a state-of-the art baseline for sequential recommendations. However, not all subsequent publications confirmed this result and proposed other models that were shown to outperform BERT4Rec in effectiveness. In this paper we systematically review all publications that compare BERT4Rec with another popular Transformer-based model, namely SASRec, and show that BERT4Rec results are not consistent within these publications. To understand the reasons behind this inconsistency, we analyse the available implementations of BERT4Rec and show that we fail to reproduce results of the original BERT4Rec publication when using their default configuration parameters. However, we are able to replicate the reported results with the original code if training for a much longer amount of time (up to 30x) compared to the default configuration. We also propose our own implementation of BERT4Rec based on the Hugging Face Transformers library, which we demonstrate replicates the originally reported results on 3 out 4 datasets, while requiring up to 95% less training time to converge. Overall, from our systematic review and detailed experiments, we conclude that BERT4Rec does indeed exhibit state-of-the-art effectiveness for sequential recommendation, but only when trained for a sufficient amount of time. Additionally, we show that our implementation can further benefit from adapting other Transformer architectures that are available in the Hugging Face Transformers library (e.g. using disentangled attention, as provided by DeBERTa, or larger hidden layer size cf. ALBERT).