# LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking

arXiv: https://arxiv.org/abs/2311.02089 · HTML: https://mlsanity.com/paper/2311.02089/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction](https://mlsanity.com/paper/2305.06474/) (2023)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [PALR: Personalization Aware LLMs for Recommendation](https://mlsanity.com/paper/2305.07622/) (2023)
- [Linear Recurrent Units for Sequential Recommendation](https://mlsanity.com/paper/2310.02367/) (2023)
- [Zero-Shot Recommendation as Language Modeling](https://mlsanity.com/paper/2112.04184/) (2021)
- [Defending Substitution-Based Profile Pollution Attacks on Sequential Recommenders](https://mlsanity.com/paper/2207.11237/) (2022)

## Built on by (most cited first, 30 of 53)

- [Position: Graph Foundation Models are Already Here](https://mlsanity.com/paper/2402.02216/) (2024)
- [A Survey on LLM-powered Agents for Recommender Systems](https://mlsanity.com/paper/2502.10050/) (2025)
- [Customizing Language Models with Instance-wise LoRA for Sequential Recommendation](https://mlsanity.com/paper/2408.10159/) (2024)
- [Rethinking Large Language Model Architectures for Sequential Recommendations](https://mlsanity.com/paper/2402.09543/) (2024)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyond](https://mlsanity.com/paper/2410.19744/) (2024)
- [Large Language Models and Video Games: A Preliminary Scoping Review](https://mlsanity.com/paper/2403.02613/) (2024)
- [RDRec: Rationale Distillation for LLM-based Recommendation](https://mlsanity.com/paper/2405.10587/) (2024)
- [Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation](https://mlsanity.com/paper/2408.06276/) (2024)
- [LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application](https://mlsanity.com/paper/2405.03988/) (2024)
- [Federated Recommendation via Hybrid Retrieval Augmented Generation](https://mlsanity.com/paper/2403.04256/) (2024)
- [A Survey of Generative Search and Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2404.16924/) (2024)
- [The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems](https://mlsanity.com/paper/2507.02097/) (2025)
- [Exploring the Impact of Large Language Models on Recommender Systems: An Extensive Review](https://mlsanity.com/paper/2402.18590/) (2024)
- [Make Large Language Model a Better Ranker](https://mlsanity.com/paper/2403.19181/) (2024)
- [CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation](https://mlsanity.com/paper/2511.22707/) (2025)
- [Reindex-Then-Adapt: Improving Large Language Models for Conversational Recommendation](https://mlsanity.com/paper/2405.12119/) (2024)
- [Enhancing High-order Interaction Awareness in LLM-based Recommender Model](https://mlsanity.com/paper/2409.19979/) (2024)
- [Token-level Proximal Policy Optimization for Query Generation](https://mlsanity.com/paper/2411.00722/) (2024)
- [Are Longer Prompts Always Better? Prompt Selection in Large Language Models for Recommendation Systems](https://mlsanity.com/paper/2412.14454/) (2024)
- [Understanding Generative Recommendation with Semantic IDs from a Model-scaling View](https://mlsanity.com/paper/2509.25522/) (2025)
- [Generative Product Recommendations for Implicit Superlative Queries](https://mlsanity.com/paper/2504.18748/) (2025)
- [DELRec: Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation](https://mlsanity.com/paper/2406.11156/) (2024)
- [MLLMRec: A Preference Reasoning Paradigm with Graph Refinement for Multimodal Recommendation](https://mlsanity.com/paper/2508.15304/) (2025)
- [Efficient and Responsible Adaptation of Large Language Models for Robust and Equitable Top-k Recommendations](https://mlsanity.com/paper/2501.04762/) (2025)
- [De-conflating Preference and Qualification: Constrained Dual-Perspective Reasoning for Job Recommendation with Large Language Models](https://mlsanity.com/paper/2602.03097/) (2026)
- [Advancing Large Language Models for Spatiotemporal and Semantic Association Mining of Similar Environmental Events](https://mlsanity.com/paper/2411.12880/) (2024)
- [PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models](https://mlsanity.com/paper/2412.05339/) (2024)
- [Do LLMs Understand Collaborative Signals? Diagnosis and Repair](https://mlsanity.com/paper/2505.20730/) (2025)
- [Augment or Not? A Comparative Study of Pure and Augmented Large Language Model Recommenders](https://mlsanity.com/paper/2505.23053/) (2025)

## Abstract (arXiv)

Recently, large language models (LLMs) have exhibited significant progress in language understanding and generation. By leveraging textual features, customized LLMs are also applied for recommendation and demonstrate improvements across diverse recommendation scenarios. Yet the majority of existing methods perform training-free recommendation that heavily relies on pretrained knowledge (e.g., movie recommendation). In addition, inference on LLMs is slow due to autoregressive generation, rendering existing methods less effective for real-time recommendation. As such, we propose a two-stage framework using large language models for ranking-based recommendation (LlamaRec). In particular, we use small-scale sequential recommenders to retrieve candidates based on the user interaction history. Then, both history and retrieved items are fed to the LLM in text via a carefully designed prompt template. Instead of generating next-item titles, we adopt a verbalizer-based approach that transforms output logits into probability distributions over the candidate items. Therefore, the proposed LlamaRec can efficiently rank items without generating long text. To validate the effectiveness of the proposed framework, we compare against state-of-the-art baseline methods on benchmark datasets. Our experimental results demonstrate the performance of LlamaRec, which consistently achieves superior performance in both recommendation performance and efficiency.