SAGE: Sequence-level Adaptive Gradient Evolution for Generative Recommendation
On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG
Abstract (arXiv)
Reinforcement learning-based preference optimization is increasingly used to align list-wise generative recommenders with complex, multi-objective user feedback, yet existing optimizers such as Gradient-Bounded Policy Optimization (GBPO) exhibit structural limitations in recommendation settings. We identify a Symmetric Conservatism failure mode in which symmetric update bounds suppress learning from rare positive signals (e.g., cold-start items), static negative-sample constraints fail to prevent diversity collapse under rejection-dominated feedback, and group-normalized multi-objective rewards lead to low-resolution training signals. To address these issues, we propose SAGE (Sequence-level Adaptive Gradient Evolution), a unified optimizer designed for list-wise generative recommendation. SAGE introduces sequence-level signal alignment via a geometric-mean importance ratio and a decoupled multi-objective advantage estimator to reduce token-level variance and mitigate reward collapse, together with asymmetric adaptive bounding that applies positive Boost updates to successful slates and an entropy-aware penalty to discourage low-diversity failures. Experiments on Amazon Product Reviews and the large-scale RecIF-Bench demonstrate consistent improvements in top-K accuracy, cold-start recall, and diversity across both Semantic-ID and native-text action spaces, while preserving numerical stability during training. These results suggest that asymmetric, sequence-aware policy optimization provides a principled and effective framework for addressing optimization failures in generative recommendation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 44 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
33 of 44 references.
Closest ancestors · 1
Same direction, newest first
Other references · 32
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2018Self-Attentive Sequential Recommendationcited by 1,184
- 2017Neural Collaborative Filteringcited by 1,111
- 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
- 2016Wide & Deep Learning for Recommender Systemscited by 747
- 2015Image-based Recommendations on Styles and Substitutescited by 706
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
- 2015Sequence Level Training with Recurrent Neural Networkscited by 576
20 more
- 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
- 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
- 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
- 2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
- 2023Recommender Systems with Generative Retrievalcited by 198
- 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
- 2021Reinforcement learning based recommender systems: A surveycited by 188
- 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
- 2023Adapting Large Language Models by Integrating Collaborative Semantics for Recommendationcited by 179
- 2025OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignmentcited by 173
- 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 103
- 2023PALR: Personalization Aware LLMs for Recommendationcited by 96
- 2023GenRec: Large Language Model for Generative Recommendationcited by 63
- 2023LLaRA: Large Language-Recommendation Assistantcited by 56
- 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 51
- 2024Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender Systemcited by 50
- 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
- 2017Deep Reinforcement Learning for List-wise Recommendationscited by 35
- 2017Cascade Ranking for Operational E-commerce Searchcited by 29
- 2024Integrating Large Language Models into Recommendation via Mutual Augmentation and Adaptive Aggregationcited by 26
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendation
- 2026Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHF
- 2026Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation
- 2026Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation
- 2026EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment
- 2026OxygenREC-v2: Internalizing Discrimination into Generative Recommendation ↗
- 2026FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning
- 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMs