Improve Long-term Memory Learning Through Rescaling the Error Temporally
Not on the map. Why?
Abstract (arXiv)
This paper studies the error metric selection for long-term memory learning in sequence modelling. We examine the bias towards short-term memory in commonly used errors, including mean absolute/squared error. Our findings show that all temporally positive-weighted errors are biased towards short-term memory in learning linear functionals. To reduce this bias and improve long-term memory learning, we propose the use of a temporally rescaled error. In addition to reducing the bias towards short-term memory, this approach can also alleviate the vanishing gradient issue. We conduct numerical experiments on different long-memory tasks and sequence models to validate our claims. Numerical results confirm the importance of appropriate temporally rescaled error for effective long-term memory learning. To the best of our knowledge, this is the first work that quantitatively analyzes different errors' memory bias towards short-term memory in sequence modelling.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 21 Jul 2023.
Builds on
30 of 42 references.
References · 30
- 2017Attention Is All You Needcited by 60,106
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,024
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,473
- 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,460
- 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 973
- 2022Simplified State Space Layers for Sequence Modelingcited by 781
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 667
- 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
- 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 602
18 more
- 2019Text Summarization with Pretrained Encoderscited by 554
- 2015Unitary Evolution Recurrent Neural Networkscited by 291
- 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 143
- 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
- 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
- 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
- 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
- 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
- 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
- 2020Do RNN and LSTM have Long Memory?cited by 47
- 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
- 2021Long Expressive Memory for Sequence Modelingcited by 41
- 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 27
- 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
- 2017Kronecker Recurrent Unitscited by 13
- 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 11
- 2021Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decaycited by 7
- 2018Short-term Memory of Deep RNNcited by 5
Built on by
Citing papers per year: 2023: 1.
1 paper.