Training Language Models Using Target-Propagation
Not on the map. Why?
Abstract (arXiv)
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps. We investigate whether Target Propagation (TPROP) style approaches can address these shortcomings. Unfortunately, extensive experiments suggest that TPROP generally underperforms BPTT, and we end with an analysis of this phenomenon, and suggestions for future work.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2017.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 2; 2022: 1; 2023: 2.
5 papers.
- 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
- 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
- 2021Backprop-Free Reinforcement Learning with Active Neural Generative Codingcited by 12
- 2022Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problemscited by 6
- 2021Target Propagation via Regularized Inversioncited by 3
- 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment1 Dec 2023
- 2023Brain-inspired Computational Intelligence via Predictive Coding15 Aug 2023
- 2022Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems19 Sep 2022
- 2021Target Propagation via Regularized Inversion2 Dec 2021
- 2021Backprop-Free Reinforcement Learning with Active Neural Generative Coding10 Jul 2021