Temporal Difference Learning with Constrained Initial Representations
Not on the map. Why?
Abstract (arXiv)
Recently, there have been numerous attempts to enhance the sample efficiency of off-policy reinforcement learning (RL) agents when interacting with the environment, including architecture improvements and new algorithms. Despite these advances, they overlook the potential of directly constraining the initial representations of the input data, which can intuitively alleviate the distribution shift issue and stabilize training. In this paper, we introduce the Tanh function into the initial layer to fulfill such a constraint. We theoretically unpack the convergence property of the temporal difference learning with the Tanh function under linear function approximation. Motivated by theoretical insights, we present our Constrained Initial Representations framework, tagged CIR, which is made up of three components: (i) the Tanh activation along with normalization methods to stabilize representations; (ii) the skip connection module to provide a linear pathway from the shallow layer to the deep layer; (iii) the convex Q-learning that allows a more flexible value estimate and mitigates potential conservatism. Empirical results show that CIR exhibits strong performance on numerous continuous control tasks, even being competitive or surpassing existing strong baseline methods.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 77 of its 93 references that are in the map. First posted on arXiv 12 Feb 2026.
Builds on
77 of 93 references.
References · 77
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2022Generative Adversarial Networkscited by 5,603
- 2016Layer Normalizationcited by 5,579
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
38 more
- 2015Prioritized Experience Replaycited by 1,070
- 2018Exploration by Random Network Distillationcited by 966
- 2018DeepMind Control Suitecited by 935
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
- 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
- 2022Temporal Difference Learning for Model Predictive Controlcited by 375
- 2024xLSTM: Extended Long Short-Term Memorycited by 320
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
- 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
- 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
- 2017Regularizing and Optimizing LSTM Language Modelscited by 205
- 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
- 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
- 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
- 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
- 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
- 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
- 2018On Regularized Losses for Weakly-supervised CNN Segmentationcited by 121
- 2018Generalization and Regularization in DQNcited by 117
- 2016Resnet in Resnet: Generalizing Residual Architecturescited by 110
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
- 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
- 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
- 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
- 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
- 2019Better Exploration with Optimistic Actor-Criticcited by 88
- 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
- 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
- 2024Simplifying Deep Temporal Difference Learningcited by 71
- 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
50 most cited of 77 All 77 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.