mlsanityData to 14 Oct 2026

ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks

arXiv 2412.12700 ↗First posted 17 Dec 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel Parallel and Modular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 33 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

7 of 33 references.

References · 7

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  4. 2018Distributed Prioritized Experience Replaycited by 269
  5. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  6. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  7. 2023SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Corescited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning
  2. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
  3. 2019Reinforcement Learning with Non-Markovian Rewards
  4. 2025Fully Learnable Neural Reward Machines
  5. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  6. 2024Neural Reward Machines
  7. 2025Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes
  8. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions