# Enhancing Recommender Systems Using Textual Embeddings from Pre-trained Language Models

arXiv: https://arxiv.org/abs/2504.08746 · HTML: https://mlsanity.com/paper/2504.08746/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction](https://mlsanity.com/paper/1910.05552/) (2019)
- [ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models](https://mlsanity.com/paper/2305.06566/) (2023)
- [RecBole 2.0: Towards a More Up-to-Date Recommendation Library](https://mlsanity.com/paper/2206.07351/) (2022)
- [EulerNet: Adaptive Feature Interaction Learning via Euler's Formula for CTR Prediction](https://mlsanity.com/paper/2304.10711/) (2023)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [Improving Semantic Similarity Measure Within a Recommender System Based-on RDF Graphs](https://mlsanity.com/paper/2307.10639/) (2023)
- [A Constraint-based Recommender System via RDF Knowledge Graphs](https://mlsanity.com/paper/2307.10702/) (2023)
- [Combining Embedding-Based and Semantic-Based Models for Post-hoc Explanations in Recommender Systems](https://mlsanity.com/paper/2401.04474/) (2024)
- [A Personalized Recommender System Based-on Knowledge Graph Embeddings](https://mlsanity.com/paper/2307.10680/) (2023)

## Abstract (arXiv)

Recent advancements in language models and pre-trained language models like BERT and RoBERTa have revolutionized natural language processing, enabling a deeper understanding of human-like language. In this paper, we explore enhancing recommender systems using textual embeddings from pre-trained language models to address the limitations of traditional recommender systems that rely solely on explicit features from users, items, and user-item interactions. By transforming structured data into natural language representations, we generate high-dimensional embeddings that capture deeper semantic relationships between users, items, and contexts. Our experiments demonstrate that this approach significantly improves recommendation accuracy and relevance, resulting in more personalized and context-aware recommendations. The findings underscore the potential of PLMs to enhance the effectiveness of recommender systems.