# Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT

arXiv: https://arxiv.org/abs/2107.00481 · HTML: https://mlsanity.com/paper/2107.00481/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Stochastic Variance-Reduced Policy Gradient](https://mlsanity.com/paper/1806.05618/) (2018)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [On the Linear Convergence of the ADMM in Decentralized Consensus Optimization](https://mlsanity.com/paper/1307.5561/) (2013)
- [Mobile Edge Computing: A Survey on Architecture and Computation Offloading](https://mlsanity.com/paper/1702.05309/) (2017)
- [Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey](https://mlsanity.com/paper/1507.01030/) (2015)
- [Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization](https://mlsanity.com/paper/1804.06568/) (2018)
- [Over-the-Air Computation Systems: Optimization, Analysis and Scaling Laws](https://mlsanity.com/paper/1909.00329/) (2019)
- [Accelerated Distributed Nesterov Gradient Descent](https://mlsanity.com/paper/1705.07176/) (2017)
- [Stochastic Recursive Momentum for Policy Gradient Methods](https://mlsanity.com/paper/2003.04302/) (2020)
- [Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems](https://mlsanity.com/paper/1809.05149/) (2018)
- [Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services](https://mlsanity.com/paper/1810.04118/) (2018)
- [Multi-agent Reinforcement Learning for Resource Allocation in IoT networks with Edge Computing](https://mlsanity.com/paper/2004.02315/) (2020)
- [Lyapunov Functions for First-Order Methods: Tight Automated Convergence Guarantees](https://mlsanity.com/paper/1803.06073/) (2018)
- [Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning](https://mlsanity.com/paper/1812.03239/) (2018)
- [Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks](https://mlsanity.com/paper/2002.11045/) (2020)
- [Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization](https://mlsanity.com/paper/2008.01296/) (2020)
- [CADA: Communication-Adaptive Distributed Adam](https://mlsanity.com/paper/2012.15469/) (2020)
- [Deep Reinforcement Learning Based Spectrum Allocation in Integrated Access and Backhaul Networks](https://mlsanity.com/paper/2004.13133/) (2020)
- [Accelerated Variance Reduced Stochastic ADMM](https://mlsanity.com/paper/1707.03190/) (2017)
- [Trusted Wireless Monitoring based on Blockchain over NB-IoT Connectivity](https://mlsanity.com/paper/2004.06299/) (2020)
- [On the Latency, Rate and Reliability Tradeoff in Wireless Networked Control Systems for IIoT](https://mlsanity.com/paper/2007.00256/) (2020)
- [Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing](https://mlsanity.com/paper/2010.00914/) (2020)
- [Managing Fog Networks using Reinforcement Learning Based Load Balancing Algorithm](https://mlsanity.com/paper/1901.10023/) (2019)
- [Privacy-preserving Incremental ADMM for Decentralized Consensus Optimization](https://mlsanity.com/paper/2003.10615/) (2020)

## Abstract (arXiv)

Edge computing provides a promising paradigm to support the implementation of Industrial Internet of Things (IIoT) by offloading tasks to nearby edge nodes. Meanwhile, the increasing network size makes it impractical for centralized data processing due to limited bandwidth, and consequently a decentralized learning scheme is preferable. Reinforcement learning (RL) has been widely investigated and shown to be a promising solution for decision-making and optimal control processes. For RL in a decentralized setup, edge nodes (agents) connected through a communication network aim to work collaboratively to find a policy to optimize the global reward as the sum of local rewards. However, communication costs, scalability and adaptation in complex environments with heterogeneous agents may significantly limit the performance of decentralized RL. Alternating direction method of multipliers (ADMM) has a structure that allows for decentralized implementation, and has shown faster convergence than gradient descent based methods. Therefore, we propose an adaptive stochastic incremental ADMM (asI-ADMM) algorithm and apply the asI-ADMM to decentralized RL with edge-computing-empowered IIoT networks. We provide convergence properties for proposed algorithms by designing a Lyapunov function and prove that the asI-ADMM has O(frac1k) +O(frac1M) convergence rate where k and  M are the number of iterations and batch samples, respectively. Then, we test our algorithm with two supervised learning problems. For performance evaluation, we simulate two applications in decentralized RL settings with homogeneous and heterogeneous agents. The experiment results show that our proposed algorithms outperform the state of the art in terms of communication costs and scalability, and can well adapt to complex IoT environments.