# GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization

arXiv: https://arxiv.org/abs/2507.14758 · HTML: https://mlsanity.com/paper/2507.14758/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in LLM reasoning for recommendation, one of 17 directions in LLM reasoning.
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Placed in

- LLM reasoning / [LLM reasoning for recommendation](https://mlsanity.com/lr/llm-reasoning-for-recommendation/)

## Builds on (most cited first, 11 of 11)

- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [Generative Sequential Recommendation with GPTRec](https://mlsanity.com/paper/2306.11114/) (2023)
- [Language Models As Semantic Indexers](https://mlsanity.com/paper/2310.07815/) (2023)
- [MMGRec: Multimodal Generative Recommendation with Transformer Model](https://mlsanity.com/paper/2404.16555/) (2024)
- [ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendation](https://mlsanity.com/paper/2502.13581/) (2025)
- [UNGER: Generative Recommendation with A Unified Code via Semantic and Collaborative Integration](https://mlsanity.com/paper/2502.06269/) (2025)

## Built on by (most cited first, 4 of 4)

- [LLMs as Orchestrators: Constraint-Compliant Multi-Agent Optimization for Recommendation Systems](https://mlsanity.com/paper/2601.19121/) (2026)
- [RobustExplain: Evaluating Robustness of LLM-Based Explanation Agents for Recommendation](https://mlsanity.com/paper/2601.19120/) (2026)
- [To See or To Read: User Behavior Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2511.03845/) (2025)
- [Dual-Context Analog Retrieval for Time Series Forecasting](https://mlsanity.com/paper/2610.03491/) (2026)

## Abstract (arXiv)

Generative models have recently demonstrated strong potential in multi-behavior recommendation systems, leveraging the expressive power of transformers and tokenization to generate personalized item sequences. However, their adoption is hindered by (1) the lack of explicit information for token reasoning, (2) high computational costs due to quadratic attention complexity and dense sequence representations after tokenization, and (3) limited multi-scale modeling over user history. In this work, we propose GRACE (Generative Recommendation via journey-aware sparse Attention on Chain-of-thought tokEnization), a novel generative framework for multi-behavior sequential recommendation. GRACE introduces a hybrid Chain-of-Thought (CoT) tokenization method that encodes user-item interactions with explicit attributes from product knowledge graphs (e.g., category, brand, price) over semantic tokenization, enabling interpretable and behavior-aligned generation. To address the inefficiency of standard attention, we design a Journey-Aware Sparse Attention (JSA) mechanism, which selectively attends to compressed, intra-, inter-, and current-context segments in the tokenized sequence. Experiments on two real-world datasets show that GRACE significantly outperforms state-of-the-art baselines, achieving up to +106.9% HR@10 and +106.7% NDCG@10 improvement over the state-of-the-art baseline on the Home domain, and +22.1% HR@10 on the Electronics domain. GRACE also reduces attention computation by up to 48% with long sequences.