# A Quantitative Comparison of Centralised and Distributed Reinforcement Learning-Based Control for Soft Robotic Arms

arXiv: https://arxiv.org/abs/2511.02192 · HTML: https://mlsanity.com/paper/2511.02192/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 13 of 13)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Hardware Conditioned Policies for Multi-Robot Transfer Learning](https://mlsanity.com/paper/1811.09864/) (2018)
- [Model Based Control of Soft Robots: A Survey of the State of the Art and Open Challenges](https://mlsanity.com/paper/2110.01358/) (2021)
- [Elastica: A compliant mechanics environment for soft robotic control](https://mlsanity.com/paper/2009.08422/) (2020)
- [Distributed Reinforcement Learning for Robot Teams: A Review](https://mlsanity.com/paper/2204.03516/) (2022)
- [Data-driven Methods Applied to Soft Robot Modeling and Control: A Review](https://mlsanity.com/paper/2305.12137/) (2023)
- [COMPOSER: Scalable and Robust Modular Policies for Snake Robots](https://mlsanity.com/paper/2310.00871/) (2023)
- [Resilient robot teams: a review integrating decentralised control, change-detection, and learning](https://mlsanity.com/paper/2204.10063/) (2022)
- [Sustainably Grown: The Underdog Robots of the Future](https://mlsanity.com/paper/2206.10306/) (2022)

## Abstract (arXiv)

This paper presents a quantitative comparison between centralised and distributed multi-agent reinforcement learning (MARL) architectures for controlling a soft robotic arm modelled as a Cosserat rod in simulation. Using PyElastica and the OpenAI Gym interface, we train both a global Proximal Policy Optimisation (PPO) controller and a Multi-Agent PPO (MAPPO) under identical budgets. Both approaches are based on the arm having n number of controlled sections. The study systematically varies n and evaluates the performance of the arm to reach a fixed target in three scenarios: default baseline condition, recovery from external disturbance, and adaptation to actuator failure. Quantitative metrics used for the evaluation are mean action magnitude, mean final distance, mean episode length, and success rate. The results show that there are no significant benefits of the distributed policy when the number of controlled sections n≤4. In very simple systems, when n≤2, the centralised policy outperforms the distributed one. When n increases to 4< n≤ 12, the distributed policy shows a high sample efficiency. In these systems, distributed policy promotes a stronger success rate, resilience, and robustness under local observability and yields faster convergence given the same sample size. However, centralised policies achieve much higher time efficiency during training as it takes much less time to train the same size of samples. These findings highlight the trade-offs between centralised and distributed policy in reinforcement learning-based control for soft robotic systems and provide actionable design guidance for future sim-to-real transfer in soft rod-like manipulators.