# Temporally Layered Architecture for Adaptive, Distributed and Continuous Control

arXiv: https://arxiv.org/abs/2301.00723 · HTML: https://mlsanity.com/paper/2301.00723/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 25 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Why should we add early exits to neural networks?](https://mlsanity.com/paper/2004.12814/) (2020)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [Delay-Aware Model-Based Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2005.05440/) (2020)
- [At Human Speed: Deep Reinforcement Learning with Action Delay](https://mlsanity.com/paper/1810.07286/) (2018)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [An Analysis of Frame-skipping in Reinforcement Learning](https://mlsanity.com/paper/2102.03718/) (2021)
- [TAAC: Temporally Abstract Actor-Critic for Continuous Control](https://mlsanity.com/paper/2104.06521/) (2021)
- [Reactive Reinforcement Learning in Asynchronous Environments](https://mlsanity.com/paper/1802.06139/) (2018)
- [Diversity-enabled sweet spots in layered architectures and speed-accuracy trade-offs in sensorimotor control](https://mlsanity.com/paper/1909.08601/) (2019)
- [Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act](https://mlsanity.com/paper/2203.08542/) (2022)

## Built on by (most cited first, 1 of 1)

- [Social Interpretable Reinforcement Learning](https://mlsanity.com/paper/2401.15480/) (2024)

## Abstract (arXiv)

We present temporally layered architecture (TLA), a biologically inspired system for temporally adaptive distributed control. TLA layers a fast and a slow controller together to achieve temporal abstraction that allows each layer to focus on a different time-scale. Our design is biologically inspired and draws on the architecture of the human brain which executes actions at different timescales depending on the environment's demands. Such distributed control design is widespread across biological systems because it increases survivability and accuracy in certain and uncertain environments. We demonstrate that TLA can provide many advantages over existing approaches, including persistent exploration, adaptive control, explainable temporal behavior, compute efficiency and distributed control. We present two different algorithms for training TLA: (a) Closed-loop control, where the fast controller is trained over a pre-trained slow controller, allowing better exploration for the fast controller and closed-loop control where the fast controller decides whether to "act-or-not" at each timestep; and (b) Partially open loop control, where the slow controller is trained over a pre-trained fast controller, allowing for open loop-control where the slow controller picks a temporally extended action or defers the next n-actions to the fast controller. We evaluated our method on a suite of continuous control tasks and demonstrate the advantages of TLA over several strong baselines.