# Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation

arXiv: https://arxiv.org/abs/2307.04571 · HTML: https://mlsanity.com/paper/2307.04571/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2023.

## Built on by (most cited first, 30 of 38)

- [Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective](https://mlsanity.com/paper/2510.10150/) (2025)
- [Cost-Effective Online Multi-LLM Selection with Versatile Reward Models](https://mlsanity.com/paper/2405.16587/) (2024)
- [How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspective](https://mlsanity.com/paper/2404.12008/) (2024)
- [SPRec: Self-Play to Debias LLM-based Recommendation](https://mlsanity.com/paper/2412.09243/) (2024)
- [Agentic Feedback Loop Modeling Improves Recommendation and User Simulation](https://mlsanity.com/paper/2410.20027/) (2024)
- [PageRank Bandits for Link Prediction](https://mlsanity.com/paper/2411.01410/) (2024)
- [Distinguished Quantized Guidance for Diffusion-based Sequence Recommendation](https://mlsanity.com/paper/2501.17670/) (2025)
- [RecGPT Technical Report](https://mlsanity.com/paper/2507.22879/) (2025)
- [On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2308.11336/) (2023)
- [Neural Contextual Bandits for Personalized Recommendation](https://mlsanity.com/paper/2312.14037/) (2023)
- [Interactive Recommendation Agent with Active User Commands](https://mlsanity.com/paper/2509.21317/) (2025)
- [EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2402.15164/) (2024)
- [Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systems](https://mlsanity.com/paper/2508.05673/) (2025)
- [Meta Clustering of Neural Bandits](https://mlsanity.com/paper/2408.05586/) (2024)
- [ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2407.13163/) (2024)
- [DECAF: Learning to be Fair in Multi-agent Resource Allocation](https://mlsanity.com/paper/2502.04281/) (2025)
- [Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems](https://mlsanity.com/paper/2403.17634/) (2024)
- [CDR: Conservative Doubly Robust Learning for Debiased Recommendation](https://mlsanity.com/paper/2308.08461/) (2023)
- [Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation](https://mlsanity.com/paper/2502.02327/) (2025)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [Advancing Loss Functions in Recommender Systems: A Comparative Study with a R'enyi Divergence-Based Solution](https://mlsanity.com/paper/2506.15120/) (2025)
- [FusionCounting: Robust visible-infrared image fusion guided by crowd counting via multi-task learning](https://mlsanity.com/paper/2508.20817/) (2025)
- [Breaking User-Centric Agency: A Tri-Party Framework for Agent-Based Recommendation](https://mlsanity.com/paper/2603.10673/) (2026)
- [When Collaborative Filtering is not Collaborative: Unfairness of PCA for Recommendations](https://mlsanity.com/paper/2310.09687/) (2023)
- [A Unified Online-Offline Framework for Co-Branding Campaign Recommendations](https://mlsanity.com/paper/2505.22254/) (2025)
- [Talos: Optimizing Top-K Accuracy in Recommender Systems](https://mlsanity.com/paper/2601.19276/) (2026)
- [LLM-Enhanced Reinforcement Learning for Long-Term User Satisfaction in Interactive Recommendation](https://mlsanity.com/paper/2601.19585/) (2026)
- [HyCoRec: Hypergraph-Enhanced Multi-Preference Learning for Alleviating Matthew Effect in Conversational Recommendation](https://mlsanity.com/paper/2607.17461/) (2026)
- [A Predict-Then-Optimize Customer Allocation Framework for Online Fund Recommendation](https://mlsanity.com/paper/2503.03165/) (2025)
- [Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2506.22112/) (2025)

## Abstract (arXiv)

Offline reinforcement learning (RL), a technology that offline learns a policy from logged data without the need to interact with online environments, has become a favorable choice in decision-making processes like interactive recommendation. Offline RL faces the value overestimation problem. To address it, existing methods employ conservatism, e.g., by constraining the learned policy to be close to behavior policies or punishing the rarely visited state-action pairs. However, when applying such offline RL to recommendation, it will cause a severe Matthew effect, i.e., the rich get richer and the poor get poorer, by promoting popular items or categories while suppressing the less popular ones. It is a notorious issue that needs to be addressed in practical recommender systems. In this paper, we aim to alleviate the Matthew effect in offline RL-based recommendation. Through theoretical analyses, we find that the conservatism of existing methods fails in pursuing users' long-term satisfaction. It inspires us to add a penalty term to relax the pessimism on states with high entropy of the logging policy and indirectly penalizes actions leading to less diverse states. This leads to the main technical contribution of the work: Debiased model-based Offline RL (DORL) method. Experiments show that DORL not only captures user interests well but also alleviates the Matthew effect. The implementation is available via https://github.com/chongminggao/DORL-codes.