mlsanityData to 14 Oct 2026

Diffusion-Based Offline RL for Improved Decision-Making in Augmented ARC Task

arXiv 2410.11324 ↗First posted 15 Oct 2024cs.AI cs.CV cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Effective long-term strategies enable AI systems to navigate complex environments by making sequential decisions over extended horizons. Similarly, reinforcement learning (RL) agents optimize decisions across sequences to maximize rewards, even without immediate feedback. To verify that Latent Diffusion-Constrained Q-learning (LDCQ), a prominent diffusion-based offline RL method, demonstrates strong reasoning abilities in multi-step decision-making, we aimed to evaluate its performance on the Abstraction and Reasoning Corpus (ARC). However, applying offline RL methodologies to enhance strategic reasoning in AI for solving tasks in ARC is challenging due to the lack of sufficient experience data in the ARC training set. To address this limitation, we introduce an augmented offline RL dataset for ARC, called Synthesized Offline Learning Data for Abstraction and Reasoning (SOLAR), along with the SOLAR-Generator, which generates diverse trajectory data based on predefined rules. SOLAR enables the application of offline RL methods by offering sufficient experience data. We synthesized SOLAR for a simple task and used it to train an agent with the LDCQ method. Our experiments demonstrate the effectiveness of the offline RL approach on a simple ARC task, showing the agent's ability to make multi-step sequential decisions and correctly identify answer states. These results highlight the potential of the offline RL approach to enhance AI's strategic reasoning capabilities.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 21 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

16 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 13

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2015Prioritized Experience Replaycited by 1,099
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  8. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  9. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 149
  10. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33
  11. 2024Addressing the Abstraction and Reasoning Corpus via Procedural Example Generationcited by 33
  12. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 27
1 more
  1. 2024ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learningcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ARCTraj: A Dataset and Benchmark of Human Reasoning Trajectories for Abstract Problem Solvingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learning
  2. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
  3. 2025Offline Reinforcement Learning with Discrete Diffusion Skills
  4. 2025Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
  5. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  6. 2024DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning