mlsanityData to 14 Oct 2026

BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

arXiv 1904.06690 ↗First posted 14 Apr 2019cs.IR cs.LGcited by 964

Not on the map. Why?

Cited by 35 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modeling users' dynamic and evolving preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks (e.g., Recurrent Neural Network) to encode users' historical interactions from left to right into hidden representations for making recommendations. Although these methods achieve satisfactory results, they often assume a rigidly ordered sequence which is not always practical. We argue that such left-to-right unidirectional architectures restrict the power of the historical sequence representations. For this purpose, we introduce a Bidirectional Encoder Representations from Transformers for sequential Recommendation (BERT4Rec). However, jointly conditioning on both left and right context in deep bidirectional model would make the training become trivial since each item can indirectly "see the target item". To address this problem, we train the bidirectional model using the Cloze task, predicting the masked items in the sequence by jointly conditioning on their left and right context. Comparing with predicting the next item at each position in a sequence, the Cloze task can produce more samples to train a more powerful bidirectional model. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

Key facts. Cited by 964 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 64; 2022: 104; 2023: 158; 2024: 183; 2025: 223; 2026: 232.

964 papers.

Sort:

By area:

  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  2. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
  3. 2023A Survey on Large Language Models for Recommendationcited by 379
  4. 2023Recommender Systems with Generative Retrievalcited by 198
  5. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  6. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 188
  7. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  8. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 142
  9. 2023RecMind: Large Language Model Powered Agent For Recommendationcited by 129
  10. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 127
  11. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  12. 2022Towards Universal Sequence Representation Learning for Recommender Systemscited by 121
38 more
  1. 2022Learning Vector-Quantized Item Representation for Transferable Sequential Recommenderscited by 112
  2. 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 103
  3. 2023How to Index Item IDs for Recommendation Foundation Modelscited by 102
  4. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  5. 2023PALR: Personalization Aware LLMs for Recommendationcited by 96
  6. 2023DiffuRec: A Diffusion Model for Sequential Recommendationcited by 93
  7. 2022Intent Contrastive Learning for Sequential Recommendationcited by 91
  8. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  9. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  10. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 76
  11. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  12. 2023Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusioncited by 74
  13. 2021Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformercited by 69
  14. 2021AutoDebias: Learning to Debias for Recommendationcited by 68
  15. 2021SimpleX: A Simple and Strong Baseline for Collaborative Filteringcited by 65
  16. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  17. 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
  18. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  19. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  20. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  21. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  22. 2022PinnerFormer: Sequence Modeling for User Representation at Pinterestcited by 57
  23. 2021Contrastive Self-supervised Sequential Recommendation with Robust Augmentationcited by 56
  24. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  25. 2023Zero-Shot Next-Item Recommendation using Large Pretrained Language Modelscited by 53
  26. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 53
  27. 2024IDGenRec: LLM-RecSys Alignment with Textual ID Learningcited by 53
  28. 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 51
  29. 2024Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender Systemcited by 50
  30. 2021Knowledge-aware Coupled Graph Neural Network for Social Recommendationcited by 49
  31. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  32. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 46
  33. 2021Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformercited by 45
  34. 2021End-to-End User Behavior Retrieval in Click-Through RatePrediction Modelcited by 45
  35. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  36. 2022Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendationcited by 42
  37. 2023DiffKG: Knowledge Graph Diffusion Model for Recommendationcited by 41
  38. 2021Knowledge-Enhanced Hierarchical Graph Transformer Network for Multi-Behavior Recommendationcited by 39

50 most cited of 964 All 964 (JSON)