# Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors

arXiv: https://arxiv.org/abs/2403.19347 · HTML: https://mlsanity.com/paper/2403.19347/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems](https://mlsanity.com/paper/2205.08084/) (2022)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09248/) (2019)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights](https://mlsanity.com/paper/2305.15036/) (2023)
- [Scaling Law for Recommendation Models: Towards General-purpose User Representations](https://mlsanity.com/paper/2111.11294/) (2021)
- [Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction](https://mlsanity.com/paper/2205.10249/) (2022)
- [Text Matching Improves Sequential Recommendation by Reducing Popularity Biases](https://mlsanity.com/paper/2308.14029/) (2023)

## Built on by (most cited first, 23 of 23)

- [On Softmax Direct Preference Optimization for Recommendation](https://mlsanity.com/paper/2406.09215/) (2024)
- [Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation](https://mlsanity.com/paper/2503.22675/) (2025)
- [LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation](https://mlsanity.com/paper/2405.20646/) (2024)
- [Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyond](https://mlsanity.com/paper/2410.19744/) (2024)
- [Language Representations Can be What Recommenders Need: Findings and Potentials](https://mlsanity.com/paper/2407.05441/) (2024)
- [Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2504.18383/) (2025)
- [EasyRec: Simple yet Effective Language Models for Recommendation](https://mlsanity.com/paper/2408.08821/) (2024)
- [Improving LLM-powered Recommendations with Personalized Information](https://mlsanity.com/paper/2502.13845/) (2025)
- [CoMaPOI: A Collaborative Multi-Agent Framework for Next POI Prediction Bridging the Gap Between Trajectory and Language](https://mlsanity.com/paper/2505.23837/) (2025)
- [Leveraging Large Language Models for Active Merchant Non-player Characters](https://mlsanity.com/paper/2412.11189/) (2024)
- [Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training](https://mlsanity.com/paper/2504.08949/) (2025)
- [Field Matters: A Lightweight LLM-enhanced Method for CTR Prediction](https://mlsanity.com/paper/2505.14057/) (2025)
- [CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction](https://mlsanity.com/paper/2508.03668/) (2025)
- [Balancing Efficiency and Effectiveness: An LLM-Infused Approach for Optimized CTR Prediction](https://mlsanity.com/paper/2412.06860/) (2024)
- [Learning Multi-Branch Cooperation for Enhanced Click-Through Rate Prediction at Taobao](https://mlsanity.com/paper/2411.13057/) (2024)
- [LLM-EDT: Large Language Model Enhanced Cross-domain Sequential Recommendation with Dual-phase Training](https://mlsanity.com/paper/2511.19931/) (2025)
- [CORONA: A Coarse-to-Fine Framework for Graph-based Recommendation with Large Language Models](https://mlsanity.com/paper/2506.17281/) (2025)
- [Personality Prediction from Life Stories using Language Models](https://mlsanity.com/paper/2506.19258/) (2025)
- [Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives](https://mlsanity.com/paper/2512.01179/) (2025)
- [GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigm](https://mlsanity.com/paper/2602.01865/) (2026)
- [Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations](https://mlsanity.com/paper/2604.18146/) (2026)
- [SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation](https://mlsanity.com/paper/2607.24025/) (2026)
- [CARA: Cognitive Adaptive Recommendation Agent](https://mlsanity.com/paper/2608.16919/) (2026)

## Abstract (arXiv)

With the rise of large language models (LLMs), recent works have leveraged LLMs to improve the performance of click-through rate (CTR) prediction. However, we argue that a critical obstacle remains in deploying LLMs for practical use: the efficiency of LLMs when processing long textual user behaviors. As user sequences grow longer, the current efficiency of LLMs is inadequate for training on billions of users and items. To break through the efficiency barrier of LLMs, we propose Behavior Aggregated Hierarchical Encoding (BAHE) to enhance the efficiency of LLM-based CTR modeling. Specifically, BAHE proposes a novel hierarchical architecture that decouples the encoding of user behaviors from inter-behavior interactions. Firstly, to prevent computational redundancy from repeated encoding of identical user behaviors, BAHE employs the LLM's pre-trained shallow layers to extract embeddings of the most granular, atomic user behaviors from extensive user sequences and stores them in the offline database. Subsequently, the deeper, trainable layers of the LLM facilitate intricate inter-behavior interactions, thereby generating comprehensive user embeddings. This separation allows the learning of high-level user representations to be independent of low-level behavior encoding, significantly reducing computational complexity. Finally, these refined user embeddings, in conjunction with correspondingly processed item embeddings, are incorporated into the CTR model to compute the CTR scores. Extensive experimental results show that BAHE reduces training time and memory by five times for CTR models using LLMs, especially with longer user sequences. BAHE has been deployed in a real-world system, allowing for daily updates of 50 million CTR data on 8 A100 GPUs, making LLMs practical for industrial CTR prediction.