# Fat-to-Thin Policy Optimization: Offline RL with Sparse Policies

arXiv: https://arxiv.org/abs/2501.14373 · HTML: https://mlsanity.com/paper/2501.14373/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 1 of its 36 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 1 of 1)

- [q-exponential family for policy optimization](https://mlsanity.com/paper/2408.07245/) (2024)

## Built on by (most cited first, 2 of 2)

- [Symmetric Behavior Regularized Policy Optimization](https://mlsanity.com/paper/2508.04225/) (2025)
- [Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners](https://mlsanity.com/paper/2607.13274/) (2026)

## Abstract (arXiv)

Sparse continuous policies are distributions that can choose some actions at random yet keep strictly zero probability for the other actions, which are radically different from the Gaussian. They have important real-world implications, e.g. in modeling safety-critical tasks like medicine. The combination of offline reinforcement learning and sparse policies provides a novel paradigm that enables learning completely from logged datasets a safety-aware sparse policy. However, sparse policies can cause difficulty with the existing offline algorithms which require evaluating actions that fall outside of the current support. In this paper, we propose the first offline policy optimization algorithm that tackles this challenge: Fat-to-Thin Policy Optimization (FtTPO). Specifically, we maintain a fat (heavy-tailed) proposal policy that effectively learns from the dataset and injects knowledge to a thin (sparse) policy, which is responsible for interacting with the environment. We instantiate FtTPO with the general q-Gaussian family that encompasses both heavy-tailed and sparse policies and verify that it performs favorably in a safety-critical treatment simulation and the standard MuJoCo suite. Our code is available at urlhttps://github.com/lingweizhu/fat2thin.