mlsanityData to 7 Oct 2026

Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs

arXiv 2101.00674 ↗First posted 3 Jan 2021cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In Recurrent Neural Networks (RNNs), encoding information in a suboptimal or erroneous way can impact the quality of representations based on later elements in the sequence and subsequently lead to wrong predictions and a worse model performance. In humans, challenging cases like garden path sentences (an instance of this being the infamous "The horse raced past the barn fell") can lead their language understanding astray. However, they are still able to correct their representation accordingly and recover when new information is encountered. Inspired by this, I propose an augmentation to standard RNNs in form of a gradient-based correction mechanism: This way I hope to enable such models to dynamically adapt their inner representation of a sentence, adding a way to correct deviations as soon as they occur. This could therefore lead to more robust models using more flexible representations, even during inference time. I conduct different experiments in the context of language modeling, where the impact of using such a mechanism is examined in detail. To this end, I look at modifications based on different kinds of time-dependent error signals and how they influence the model performance. Furthermore, this work contains a study of the model's confidence in its predictions during training and for challenging test samples and the effect of the manipulation thereof. Lastly, I also study the difference in behavior of these novel models compared to a standard LSTM baseline and investigate error cases in detail to identify points of future research. I show that while the proposed approach comes with promising theoretical guarantees and an appealing intuition, it is only able to produce minor improvements over the baseline due to challenges in its practical application and the efficacy of the tested model variants.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 94 references that are in the map. First posted on arXiv 3 Jan 2021.

Builds on

40 of 94 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
28 more
  1. 2016An overview of gradient descent optimization algorithmscited by 1,021
  2. 2019Green AIcited by 620
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  4. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  5. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  6. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  7. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  8. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  9. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  10. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  11. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  12. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  13. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  14. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  15. 2017Building machines that adapt and compute like brainscited by 99
  16. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  17. 2017Learning to Remember Rare Eventscited by 63
  18. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  19. 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 51
  20. 2017Deep and Confident Prediction for Time Series at Ubercited by 50
  21. 2018FRAGE: Frequency-Agnostic Word Representationcited by 47
  22. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  23. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  24. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 33
  25. 2018Lipschitz regularized Deep Neural Networks generalize and are adversarially robustcited by 19
  26. 2018Polynomial Regression As an Alternative to Neural Netscited by 15
  27. 2019Finding Syntactic Representations in Neural Stackscited by 4
  28. 2018LSTMs Exploit Linguistic Attributes of Datacited by 3

Built on by

Not cited yet.