# Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

arXiv: https://arxiv.org/abs/2408.10159 · HTML: https://mlsanity.com/paper/2408.10159/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in MoE for recommender systems, one of 12 directions in Mixture-of-experts.
- Builds on 50 of its 63 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Placed in

- Mixture-of-experts / [MoE for recommender systems](https://mlsanity.com/moe/moe-for-recommender-systems/)

## Builds on (most cited first, 30 of 50)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](https://mlsanity.com/paper/2206.02770/) (2022)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)

## Built on by (most cited first, 30 of 32)

- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [Unified Parameter-Efficient Unlearning for LLMs](https://mlsanity.com/paper/2412.00383/) (2024)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steering](https://mlsanity.com/paper/2507.04756/) (2025)
- [Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives](https://mlsanity.com/paper/2508.02853/) (2025)
- [AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment](https://mlsanity.com/paper/2603.26680/) (2026)
- [A Survey on Large Language Models in Multimodal Recommender Systems](https://mlsanity.com/paper/2505.09777/) (2025)
- [Large Language Model as Universal Retriever in Industrial-Scale Recommender System](https://mlsanity.com/paper/2502.03041/) (2025)
- [Addressing Missing Data Issue for Diffusion-based Recommendation](https://mlsanity.com/paper/2505.12283/) (2025)
- [EXACT: Explicit Attribute-Guided Decoding-Time Personalization](https://mlsanity.com/paper/2602.17695/) (2026)
- [Optimization-Inspired Few-Shot Adaptation for Large Language Models](https://mlsanity.com/paper/2505.19107/) (2025)
- [CARD: Cluster-level Adaptation with Reward-guided Decoding for Personalized Text Generation](https://mlsanity.com/paper/2601.06352/) (2026)
- [Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory](https://mlsanity.com/paper/2607.00017/) (2026)
- [Augment or Not? A Comparative Study of Pure and Augmented Large Language Model Recommenders](https://mlsanity.com/paper/2505.23053/) (2025)
- [FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic](https://mlsanity.com/paper/2510.24061/) (2025)
- [RAIE: Region-Aware Incremental Preference Editing with LoRA for LLM-based Recommendation](https://mlsanity.com/paper/2603.00638/) (2026)
- [LATTE: Forecasting Peer Anchored Preference Trajectories for Personalized LLM Generation](https://mlsanity.com/paper/2605.26612/) (2026)
- [Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2501.13344/) (2025)
- [A Text-Based Recommender System that Leverages Explicit Affective State Preferences](https://mlsanity.com/paper/2505.20190/) (2025)
- [CORONA: A Coarse-to-Fine Framework for Graph-based Recommendation with Large Language Models](https://mlsanity.com/paper/2506.17281/) (2025)
- [On Negative-aware Preference Optimization for Recommendation](https://mlsanity.com/paper/2508.09653/) (2025)
- [Beyond Semantic Understanding: Preserving Collaborative Frequency Components in LLM-based Recommendation](https://mlsanity.com/paper/2508.10312/) (2025)
- [Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving](https://mlsanity.com/paper/2511.00101/) (2025)
- [Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMs](https://mlsanity.com/paper/2601.06362/) (2026)
- [Enhancing guidance for missing data in diffusion-based sequential recommendation](https://mlsanity.com/paper/2601.15673/) (2026)
- [The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage](https://mlsanity.com/paper/2602.10339/) (2026)
- [DynamicPO: Dynamic Preference Optimization for Recommendation](https://mlsanity.com/paper/2605.00327/) (2026)
- [An Agentic LLM-Based Framework for Population-Scale Mental Health Screening](https://mlsanity.com/paper/2605.13046/) (2026)
- [Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving](https://mlsanity.com/paper/2608.03579/) (2026)

## Abstract (arXiv)

Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches are eager to apply LLMs to sequential recommendation. A common paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform application of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce a sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parameter adjustment for instance-wise recommendations. In sequential recommendation, iLoRA achieves an average relative improvement of 11.4% over basic LoRA in the hit ratio metric, with less than a 1% relative increase in trainable parameters. Extensive experiments on three benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy. Our data and code are available at https://github.com/AkaliKong/iLoRA.