# HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling

arXiv: https://arxiv.org/abs/2409.12740 · HTML: https://mlsanity.com/paper/2409.12740/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Representation Learning with Large Language Models for Recommendation](https://mlsanity.com/paper/2310.15950/) (2023)
- [M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems](https://mlsanity.com/paper/2205.08084/) (2022)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction](https://mlsanity.com/paper/2305.06474/) (2023)
- [PALR: Personalization Aware LLMs for Recommendation](https://mlsanity.com/paper/2305.07622/) (2023)
- [User-LLM: Efficient LLM Contextualization with User Embeddings](https://mlsanity.com/paper/2402.13598/) (2024)
- [Large Language Models for Generative Recommendation: A Survey and Visionary Discussions](https://mlsanity.com/paper/2309.01157/) (2023)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [Scaling Law for Recommendation Models: Towards General-purpose User Representations](https://mlsanity.com/paper/2111.11294/) (2021)
- [CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendation](https://mlsanity.com/paper/2405.02429/) (2024)
- [ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest](https://mlsanity.com/paper/2205.11728/) (2022)
- [LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application](https://mlsanity.com/paper/2405.03988/) (2024)

## Built on by (most cited first, 30 of 63)

- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendation](https://mlsanity.com/paper/2506.21579/) (2025)
- [MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation](https://mlsanity.com/paper/2510.24431/) (2025)
- [R2ec: Towards Large Recommender Models with Reasoning](https://mlsanity.com/paper/2505.16994/) (2025)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Large Language Model Enhanced Recommender Systems: A Survey](https://mlsanity.com/paper/2412.13432/) (2024)
- [Improving LLM-powered Recommendations with Personalized Information](https://mlsanity.com/paper/2502.13845/) (2025)
- [Molar: Multimodal LLMs with Collaborative Filtering Alignment for Enhanced Sequential Recommendation](https://mlsanity.com/paper/2412.18176/) (2024)
- [TBGRecall: A Generative Retrieval Model for E-commerce Recommendation Scenarios](https://mlsanity.com/paper/2508.11977/) (2025)
- [LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation](https://mlsanity.com/paper/2601.20083/) (2026)
- [Large Language Model as Universal Retriever in Industrial-Scale Recommender System](https://mlsanity.com/paper/2502.03041/) (2025)
- [Field Matters: A Lightweight LLM-enhanced Method for CTR Prediction](https://mlsanity.com/paper/2505.14057/) (2025)
- [PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling](https://mlsanity.com/paper/2510.10102/) (2025)
- [SessionRec: Next Session Prediction Paradigm For Generative Sequential Recommendation](https://mlsanity.com/paper/2502.10157/) (2025)
- [Rethinking LLM-Based Recommendations: A Personalized Query-Driven Parallel Integration](https://mlsanity.com/paper/2504.11889/) (2025)
- [GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Models](https://mlsanity.com/paper/2507.06507/) (2025)
- [HLLM-Creator: Hierarchical LLM-based Personalized Creative Generation](https://mlsanity.com/paper/2508.18118/) (2025)
- [xLLM Technical Report](https://mlsanity.com/paper/2510.14686/) (2025)
- [Don't Waste It: Guiding Generative Recommenders with Structured Human Priors via Multi-Head Decoding](https://mlsanity.com/paper/2511.10492/) (2025)
- [From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction](https://mlsanity.com/paper/2511.12081/) (2025)
- [End-to-End Semantic ID Generation for Generative Advertisement Recommendation](https://mlsanity.com/paper/2602.10445/) (2026)
- [Who You Are Matters: Bridging Topics and Social Roles via LLM-Enhanced Logical Recommendation](https://mlsanity.com/paper/2505.10940/) (2025)
- [Spacetime-GR: A Spacetime-Aware Generative Model for Large Scale Online POI Recommendation](https://mlsanity.com/paper/2508.16126/) (2025)
- [SynerGen: Contextualized Generative Recommender for Unified Search and Recommendation](https://mlsanity.com/paper/2509.21777/) (2025)
- [RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference](https://mlsanity.com/paper/2601.01712/) (2026)
- [Reasoning Over Space: Enabling Geographic Reasoning for LLM-Based Generative Next POI Recommendation](https://mlsanity.com/paper/2601.04562/) (2026)
- [MuonRec: Shifting the Optimizer Paradigm Beyond Adam in Scalable Generative Recommendation](https://mlsanity.com/paper/2603.00416/) (2026)
- [KARMA: Knowledge-Action Regularized Multimodal Alignment for Personalized Search at Taobao](https://mlsanity.com/paper/2603.22779/) (2026)
- [MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches](https://mlsanity.com/paper/2604.22881/) (2026)
- [FOUNDv2: Learning Unified User Quantized Tokenizers for User Representation](https://mlsanity.com/paper/2508.00956/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success in various fields, prompting several studies to explore their potential in recommendation systems. However, these attempts have so far resulted in only modest improvements over traditional recommendation models. Moreover, three critical questions remain under-explored: firstly, the real value of LLMs' pre-trained weights, often considered to encapsulate world knowledge; secondly, the necessity of fine-tuning for recommendation tasks; lastly, whether LLMs can exhibit the same scalability benefits in recommendation systems as they do in other domains. In this paper, we propose a novel Hierarchical Large Language Model (HLLM) architecture designed to enhance sequential recommendation systems. Our approach employs a two-tier model: the first Item LLM extracts rich content features from the detailed text description of the item, while the second User LLM utilizes these features to predict users' future interests based on their interaction history. Extensive experiments demonstrate that our method effectively leverages the pre-trained capabilities of open-source LLMs, and further fine-tuning leads to significant performance boosts. Additionally, HLLM achieves excellent scalability, with the largest configuration utilizing 7B parameters for both item feature extraction and user interest modeling. Moreover, HLLM offers excellent training and serving efficiency, making it practical in real-world applications. Evaluations on two large-scale datasets, PixelRec and Amazon Reviews, show that HLLM achieves state-of-the-art results, outperforming traditional ID-based models by a wide margin. In online A/B testing, HLLM showcases notable gains, validating its practical impact in real-world recommendation scenarios. Codes are available at https://github.com/bytedance/HLLM.