# UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation

arXiv: https://arxiv.org/abs/2305.15756 · HTML: https://mlsanity.com/paper/2305.15756/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Empowering News Recommendation with Pre-trained Language Models](https://mlsanity.com/paper/2104.07413/) (2021)
- [HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation](https://mlsanity.com/paper/2106.04408/) (2021)
- [QuoteR: A Benchmark of Quote Recommendation for Writing](https://mlsanity.com/paper/2202.13145/) (2022)
- [Quotation Recommendation and Interpretation Based on Transformation from Queries to Quotations](https://mlsanity.com/paper/2105.14189/) (2021)

## Built on by (most cited first, 20 of 20)

- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2308.11131/) (2023)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction](https://mlsanity.com/paper/2310.09234/) (2023)
- [Language Representations Can be What Recommenders Need: Findings and Potentials](https://mlsanity.com/paper/2407.05441/) (2024)
- [Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation](https://mlsanity.com/paper/2408.03533/) (2024)
- [Preference-Conditioned Language-Guided Abstraction](https://mlsanity.com/paper/2402.03081/) (2024)
- [SPAR: Personalized Content-Based Recommendation via Long Engagement Attention](https://mlsanity.com/paper/2402.10555/) (2024)
- [EmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations](https://mlsanity.com/paper/2405.11441/) (2024)
- [Large Language Models Make Sample-Efficient Recommender Systems](https://mlsanity.com/paper/2406.02368/) (2024)
- [Enhancing ID-based Recommendation with Large Language Models](https://mlsanity.com/paper/2411.02041/) (2024)
- [A Survey on LLM-based News Recommender Systems](https://mlsanity.com/paper/2502.09797/) (2025)
- [Efficient Pointwise-Pairwise Learning-to-Rank for News Recommendation](https://mlsanity.com/paper/2409.17711/) (2024)
- [PrivacyXray: Detecting Privacy Breaches in LLMs through Semantic Consistency and Probability Certainty](https://mlsanity.com/paper/2506.19563/) (2025)
- [Large Language Model Driven Recommendation](https://mlsanity.com/paper/2408.10946/) (2024)
- [Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2501.13344/) (2025)
- [Stochastic Primal-Dual Decoding for Multiobjective Generative Recommender Systems](https://mlsanity.com/paper/2607.19357/) (2026)
- [GrocLM: Grocery Category Recommendation in E-Commerce with Large Language Models](https://mlsanity.com/paper/2607.24764/) (2026)

## Abstract (arXiv)

Prior study has shown that pretrained language models (PLM) can boost the performance of text-based recommendation. In contrast to previous works that either use PLM to encode user history as a whole input text, or impose an additional aggregation network to fuse multi-turn history representations, we propose a unified local- and global-attention Transformer encoder to better model two-level contexts of user history. Moreover, conditioned on user history encoded by Transformer encoders, our framework leverages Transformer decoders to estimate the language perplexity of candidate text items, which can serve as a straightforward yet significant contrastive signal for user-item text matching. Based on this, our framework, UniTRec, unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation. Extensive evaluation shows that UniTRec delivers SOTA performance on three text-based recommendation tasks. Code is available at https://github.com/Veason-silverbullet/UniTRec.