mlsanityData to 14 Oct 2026

Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns

arXiv 2503.03660 ↗First posted 5 Mar 2025cs.LGcited by 16

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We introduce a sequence-conditioned critic for Soft Actor-Critic (SAC) that models trajectory context with a lightweight Transformer and trains on aggregated N-step targets. Unlike prior approaches that (i) score state-action pairs in isolation or (ii) rely on actor-side action chunking to handle long horizons, our method strengthens the critic itself by conditioning on short trajectory segments and integrating multi-step returns -- without importance sampling (IS). The resulting sequence-aware value estimates capture the critical temporal structure for extended-horizon and sparse-reward problems. On local-motion benchmarks, we further show that freezing critic parameters for several steps makes our update compatible with CrossQ's core idea, enabling stable training without a target network. Despite its simplicity -- a 2-layer Transformer with 128-256 hidden units and a maximum update-to-data ratio (UTD) of 1 -- the approach consistently outperforms standard SAC and strong off-policy baselines, with particularly large gains on long-trajectory control. These results highlight the value of sequence modeling and N-step bootstrapping on the critic side for long-horizon reinforcement learning.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 51 of its 80 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

51 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Averaging n-step Returns Reduces Variance in Reinforcement Learning6 Feb

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
38 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  13. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  14. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  15. 2017Parameter Space Noise for Explorationcited by 161
  16. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  17. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
  18. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  19. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  20. 2024Simplifying Deep Temporal Difference Learningcited by 78
  21. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  22. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
  23. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 54
  24. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 39
  25. 2022ProDMPs: A Unified Perspective on Dynamic and Probabilistic Movement Primitivescited by 25
  26. 2022Deep Black-Box Reinforcement Learning with Movement Primitivescited by 20
  27. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 19
  28. 2021An Analysis of Frame-skipping in Reinforcement Learningcited by 18
  29. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  30. 2023Why Target Networks Stabilise Temporal Difference Methodscited by 13
  31. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  32. 2024Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learningcited by 12
  33. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  34. 2023MP3: Movement Primitive-Based (Re-)Planning Policycited by 8
  35. 2021Bridging the Gap Between Target Networks and Functional Regularizationcited by 7
  36. 2017Uncertainty Averse Pushing with Model Predictive Path Integral Controlcited by 5
  37. 2019Deep Reinforcement Learning with Decorrelationcited by 3
  38. 2024Learning the Target Network in Function Spacecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 107
  2. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 37
  3. 2025Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Rewardcited by 13
  4. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 10
  5. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 8
  6. 2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
  7. 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimationcited by 4
  8. 2025ReaCritic: Reasoning Transformer-based DRL Critic-model Scaling For Wireless Networkscited by 1
  9. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learningcited by 1
  10. 2026PAC-ACT: Post-training Actor-Critic for Action Chunking Transformerscited by 1
  11. 2025Sequence Aware SAC Control for Engine Fuel Consumption Optimization in Electrified Powertraincited by 0
  12. 2026Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllerscited by 0
4 more
  1. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  2. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  3. 2026Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learningcited by 0
  4. 2026Action Chunking Proximal Policy Optimization with Feedback Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradients
  2. 2026Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive
  3. 2026DAMPER: Return-Prioritized Gradient Control for Smooth Policies
  4. 2026A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning
  5. 2024Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization
  6. 2026Act More, Decide Less: Skill-Guided Adaptive Action Chunking for Long-Horizon LLM Agents
  7. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
  8. 2026Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control