# DiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shifts

arXiv: https://arxiv.org/abs/2510.00358 · HTML: https://mlsanity.com/paper/2510.00358/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 14 of 14)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning](https://mlsanity.com/paper/2103.06326/) (2021)
- [Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2209.06620/) (2022)
- [DROPO: Sim-to-Real Transfer with Offline Domain Randomization](https://mlsanity.com/paper/2201.08434/) (2022)
- [Data-driven Methods Applied to Soft Robot Modeling and Control: A Review](https://mlsanity.com/paper/2305.12137/) (2023)
- [Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage](https://mlsanity.com/paper/2310.18434/) (2023)
- [Reinforcement Learning Controllers for Soft Robots using Learned Environments](https://mlsanity.com/paper/2410.18519/) (2024)
- [Back-stepping Experience Replay with Application to Model-free Reinforcement Learning for a Soft Snake Robot](https://mlsanity.com/paper/2401.11372/) (2024)
- [Offline Goal-Conditioned Reinforcement Learning for Shape Control of Deformable Linear Objects](https://mlsanity.com/paper/2403.10290/) (2024)
- [Lateral oscillation and body compliance help snakes and snake robots stably traverse large, smooth obstacles](https://mlsanity.com/paper/2003.13733/) (2020)

## Abstract (arXiv)

Soft snake robots offer remarkable flexibility and adaptability in complex environments, yet their control remains challenging due to highly nonlinear dynamics. Existing model-based and bio-inspired controllers rely on simplified assumptions that limit their performance. Deep reinforcement learning (DRL) has recently emerged as a promising alternative, but online training is often impractical because of costly and potentially damaging real-world interactions. Offline RL provides a safer option by leveraging pre-collected datasets, but it suffers from distribution shift, which degrades generalization to unseen scenarios. To overcome this challenge, we propose DiSA-IQL (Distribution-Shift-Aware Implicit Q-Learning), an extension of IQL that incorporates robustness modulation by penalizing unreliable state-action pairs to mitigate distribution shift. We evaluate DiSA-IQL on goal-reaching tasks across two settings: in-distribution and out-of-distribution evaluation. Simulation results show that DiSA-IQL consistently outperforms baseline models, including Behavior Cloning (BC), Conservative Q-Learning (CQL), and vanilla IQL, achieving higher success rates, smoother trajectories, and greater robustness.