# On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems

arXiv: https://arxiv.org/abs/2407.13091 · HTML: https://mlsanity.com/paper/2407.13091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [Causal Intervention for Leveraging Popularity Bias in Recommendation](https://mlsanity.com/paper/2105.06067/) (2021)
- [Reinforcement Knowledge Graph Reasoning for Explainable Recommendation](https://mlsanity.com/paper/1906.05237/) (2019)
- [Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning](https://mlsanity.com/paper/1805.10000/) (2018)
- [Causal Embeddings for Recommendation](https://mlsanity.com/paper/1706.07639/) (2017)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [Counterfactual Explainable Recommendation](https://mlsanity.com/paper/2108.10539/) (2021)
- [Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue](https://mlsanity.com/paper/2009.09945/) (2020)
- [CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation](https://mlsanity.com/paper/2109.05261/) (2021)
- [Action-Sufficient State Representation Learning for Control with Structural Constraints](https://mlsanity.com/paper/2110.05721/) (2021)
- [Large-scale Interactive Recommendation with Tree-structured Policy Gradient](https://mlsanity.com/paper/1811.05869/) (2018)
- [Knowledge-guided Deep Reinforcement Learning for Interactive Recommendation](https://mlsanity.com/paper/2004.08068/) (2020)
- [Generative Inverse Deep Reinforcement Learning for Online Recommendation](https://mlsanity.com/paper/2011.02248/) (2020)
- [Intrinsically Motivated Reinforcement Learning based Recommendation with Counterfactual Data Augmentation](https://mlsanity.com/paper/2209.08228/) (2022)
- [Plug-and-Play Model-Agnostic Counterfactual Policy Synthesis for Deep Reinforcement Learning based Recommendation](https://mlsanity.com/paper/2208.05142/) (2022)
- [Maximum-Entropy Regularized Decision Transformer with Reward Relabelling for Dynamic Recommendation](https://mlsanity.com/paper/2406.00725/) (2024)

## Built on by (most cited first, 1 of 1)

- [Toward World Models for Epidemiology](https://mlsanity.com/paper/2604.09519/) (2026)

## Abstract (arXiv)

In Reinforcement Learning-based Recommender Systems (RLRS), the complexity and dynamism of user interactions often result in high-dimensional and noisy state spaces, making it challenging to discern which aspects of the state are truly influential in driving the decision-making process. This issue is exacerbated by the evolving nature of user preferences and behaviors, requiring the recommender system to adaptively focus on the most relevant information for decision-making while preserving generaliability. To tackle this problem, we introduce an innovative causal approach for decomposing the state and extracting Causal-InDispensable State Representations (CIDS) in RLRS. Our method concentrates on identifying the Directly Action-Influenced State Variables (DAIS) and Action-Influence Ancestors (AIA), which are essential for making effective recommendations. By leveraging conditional mutual information, we develop a framework that not only discerns the causal relationships within the generative process but also isolates critical state variables from the typically dense and high-dimensional state representations. We provide theoretical evidence for the identifiability of these variables. Then, by making use of the identified causal relationship, we construct causal-indispensable state representations, enabling the training of policies over a more advantageous subset of the agent's state space. We demonstrate the efficacy of our approach through extensive experiments, showcasing our method outperforms state-of-the-art methods.