# Adaptively Truncating Backpropagation Through Time to Control Gradient Bias

arXiv: https://arxiv.org/abs/1905.07473 · HTML: https://mlsanity.com/paper/1905.07473/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2019.

## Built on by (most cited first, 22 of 22)

- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble](https://mlsanity.com/paper/2408.04739/) (2024)
- [Expert System Gradient Descent Style Training: Development of a Defensible Artificial Intelligence Technique](https://mlsanity.com/paper/2103.04314/) (2021)
- [Prediction of the Position of External Markers Using a Recurrent Neural Network Trained With Unbiased Online Recurrent Optimization for Safe Lung Cancer Radiotherapy](https://mlsanity.com/paper/2106.01100/) (2021)
- [Efficient Training of Spiking Neural Networks with Temporally-Truncated Local Backpropagation through Time](https://mlsanity.com/paper/2201.07210/) (2021)
- [FDNet: Focal Decomposed Network for Efficient, Robust and Practical Time Series Forecasting](https://mlsanity.com/paper/2306.10703/) (2023)
- [Unifying back-propagation and forward-forward algorithms through model predictive control](https://mlsanity.com/paper/2409.19561/) (2024)
- [A Deep Learning approach for parametrized and time dependent Partial Differential Equations using Dimensionality Reduction and Neural ODEs](https://mlsanity.com/paper/2502.08683/) (2025)
- [LORD: Lower-Dimensional Embedding of Log-Signature in Neural Rough Differential Equations](https://mlsanity.com/paper/2204.08781/) (2022)
- [Tuning the burn-in phase in training recurrent neural networks improves their performance](https://mlsanity.com/paper/2602.10911/) (2026)
- [ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation](https://mlsanity.com/paper/2603.17812/) (2026)
- [Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms](https://mlsanity.com/paper/2603.20891/) (2026)
- [Steering Recurrent Reasoners at Inference Time with Readout Feedback](https://mlsanity.com/paper/2608.24136/) (2026)
- [Determining Sentencing Recommendations and Patentability Using a Machine Learning Trained Expert System](https://mlsanity.com/paper/2108.04088/) (2021)
- [Fake News and Phishing Detection Using a Machine Learning Trained Expert System](https://mlsanity.com/paper/2108.08264/) (2021)
- [Automating the Design and Development of Gradient Descent Trained Expert System Networks](https://mlsanity.com/paper/2207.02845/) (2022)
- [A Novel Paradigm for Neural Computation: X-Net with Learnable Neurons and Adaptable Structure](https://mlsanity.com/paper/2401.01772/) (2024)
- [Time-series Initialization and Conditioning for Video-agnostic Stabilization of Video Super-Resolution using Recurrent Networks](https://mlsanity.com/paper/2403.15832/) (2024)
- [Implementation and Evaluation of a Gradient Descent-Trained Defensible Blackboard Architecture System](https://mlsanity.com/paper/2404.11714/) (2024)
- [Vision Tiny Recursion Model (ViTRM): Parameter-Efficient Image Classification via Recursive State Refinement](https://mlsanity.com/paper/2603.19503/) (2026)
- [When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observability](https://mlsanity.com/paper/2609.33169/) (2026)

## Abstract (arXiv)

Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge if the truncation length is too small, or will converge slowly if it is too large. We propose an adaptive TBPTT scheme that converts the problem from choosing a temporal lag to one of choosing a tolerable amount of gradient bias. For many realistic RNNs, the TBPTT gradients decay geometrically in expectation for large lags; under this condition, we can control the bias by varying the truncation length adaptively. For RNNs with smooth activation functions, we prove that this bias controls the convergence rate of SGD with biased gradients for our non-convex loss. Using this theory, we develop a practical method for adaptively estimating the truncation length during training. We evaluate our adaptive TBPTT method on synthetic data and language modeling tasks and find that our adaptive TBPTT ameliorates the computational pitfalls of fixed TBPTT.