# LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders

arXiv: https://arxiv.org/abs/2505.04421 · HTML: https://mlsanity.com/paper/2505.04421/

## Key facts

- Cited by 46 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 7 May 2025.

## Built on by (most cited first, 30 of 46)

- [Position: The Real Barrier to LLM Agent Usability is Agentic ROI](https://mlsanity.com/paper/2505.17767/) (2025)
- [Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design](https://mlsanity.com/paper/2602.10016/) (2026)
- [Request-Only Optimization for Recommendation Systems](https://mlsanity.com/paper/2508.05640/) (2025)
- [LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation](https://mlsanity.com/paper/2601.20083/) (2026)
- [Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deployment](https://mlsanity.com/paper/2508.02929/) (2025)
- [RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment](https://mlsanity.com/paper/2602.00682/) (2026)
- [Beyond Quality: Unlocking Diversity in Ad Headline Generation with Large Language Models](https://mlsanity.com/paper/2508.18739/) (2025)
- [MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation](https://mlsanity.com/paper/2510.15286/) (2025)
- [End-to-End Semantic ID Generation for Generative Advertisement Recommendation](https://mlsanity.com/paper/2602.10445/) (2026)
- [HiSAC: Hierarchical Sparse Activation Compression for Ultra-long Sequence Modeling in Recommenders](https://mlsanity.com/paper/2602.21009/) (2026)
- [TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds](https://mlsanity.com/paper/2604.13737/) (2026)
- [RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems](https://mlsanity.com/paper/2607.29241/) (2026)
- [Efficient Sequential Recommendation for Long Term User Interest Via Personalization](https://mlsanity.com/paper/2601.03479/) (2026)
- [Compute Only Once: UG-Separation for Efficient Large Recommendation Models](https://mlsanity.com/paper/2602.10455/) (2026)
- [RQ-Reg: A Residual-Quantization-Based Framework for Continuous Value Prediction in Recommender Systems](https://mlsanity.com/paper/2602.23012/) (2026)
- [Token Factory: Efficiently Integrating Diverse Signals into Large Recommendation Models](https://mlsanity.com/paper/2606.19635/) (2026)
- [TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems](https://mlsanity.com/paper/2606.25147/) (2026)
- [SlimPer: Make Personalization Model Slim and Smart](https://mlsanity.com/paper/2607.12281/) (2026)
- [UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation](https://mlsanity.com/paper/2608.16797/) (2026)
- [Large Foundation Model for Ads Recommendation](https://mlsanity.com/paper/2508.14948/) (2025)
- [Adaptive User Interest Modeling via Conditioned Denoising Diffusion For Click-Through Rate Prediction](https://mlsanity.com/paper/2509.19876/) (2025)
- [ReaSeq: Unleashing World Knowledge via Reasoning for Sequential Modeling](https://mlsanity.com/paper/2512.21257/) (2025)
- [MALLOC: Benchmarking the Memory-aware Long Sequence Compression for Large Sequential Recommendation](https://mlsanity.com/paper/2601.20234/) (2026)
- [GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigm](https://mlsanity.com/paper/2602.01865/) (2026)
- [MDL: A Unified Multi-Distribution Learner in Large-scale Industrial Recommendation through Tokenization](https://mlsanity.com/paper/2602.07520/) (2026)
- [SOLAR: SVD-Optimized Lifelong Attention for Recommendation](https://mlsanity.com/paper/2603.02561/) (2026)
- [Retrieve-then-Adapt: Retrieval-Augmented Test-Time Adaptation for Sequential Recommendation](https://mlsanity.com/paper/2604.05379/) (2026)
- [Deep Interest Mining for Intent-Enriched Semantic IDs in Multimodal Generative Recommendation](https://mlsanity.com/paper/2604.20861/) (2026)
- [Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale](https://mlsanity.com/paper/2604.24806/) (2026)
- [When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving](https://mlsanity.com/paper/2605.04450/) (2026)

## Abstract (arXiv)

Modeling ultra-long user behavior sequences is critical for capturing both long- and short-term preferences in industrial recommender systems. Existing solutions typically rely on two-stage retrieval or indirect modeling paradigms, incuring upstream-downstream inconsistency and computational inefficiency. In this paper, we present LONGER, a Long-sequence Optimized traNsformer for GPU-Efficient Recommenders. LONGER incorporates (i) a global token mechanism for stabilizing attention over long contexts, (ii) a token merge module with lightweight InnerTransformers and hybrid attention strategy to reduce quadratic complexity, and (iii) a series of engineering optimizations, including training with mixed-precision and activation recomputation, KV cache serving, and the fully synchronous model training and serving framework for unified GPU-based dense and sparse parameter updates. LONGER consistently outperforms strong baselines in both offline metrics and online A/B testing in both advertising and e-commerce services at ByteDance, validating its consistent effectiveness and industrial-level scaling laws. Currently, LONGER has been fully deployed at more than 10 influential scenarios at ByteDance, serving billion users.