mlsanityData to 14 Oct 2026

Extending Memory for Language Modelling

arXiv 2305.11462 ↗First posted 19 May 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Breakthroughs in deep learning and memory networks have made major advances in natural language understanding. Language is sequential and information carried through the sequence can be captured through memory networks. Learning the sequence is one of the key aspects in learning the language. However, memory networks are not capable of holding infinitely long sequences in their memories and are limited by various constraints such as the vanishing or exploding gradient problem. Therefore, natural language understanding models are affected when presented with long sequential text. We introduce Long Term Memory network (LTM) to learn from infinitely long sequences. LTM gives priority to the current inputs to allow it to have a high impact. Language modeling is an important factor in natural language understanding. LTM was tested in language modeling, which requires long term memory. LTM is tested on Penn Tree bank dataset, Google Billion Word dataset and WikiText-2 dataset. We compare LTM with other language models which require long term memory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

39 of 65 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2016Language Modeling with Gated Convolutional Networkscited by 702
  8. 2015LSTM: A Search Space Odysseycited by 316
  9. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  10. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  11. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  12. 2014Recurrent Neural Network Regularizationcited by 189
27 more
  1. 2016Exploring the Limits of Language Modelingcited by 173
  2. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  3. 2015Character-Aware Neural Language Modelscited by 132
  4. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  5. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  6. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 92
  7. 2016Improving Neural Language Models with a Continuous Cachecited by 82
  8. 2016Efficient softmax approximation for GPUscited by 67
  9. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  10. 2020Do RNN and LSTM have Long Memory?cited by 47
  11. 2016Recurrent Highway Networkscited by 37
  12. 2019Language Models with Transformerscited by 36
  13. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 29
  14. 2016Multiplicative LSTM for sequence modellingcited by 27
  15. 2020Multi-task learning for natural language processing in the 2020s: where are we going?cited by 27
  16. 2019Towards Non-saturating Recurrent Units for Modelling Long-term Dependenciescited by 15
  17. 2017Factorization tricks for LSTM networkscited by 13
  18. 2014Learning Longer Memory in Recurrent Neural Networkscited by 12
  19. 2020Context based Text-generation using LSTM networkscited by 10
  20. 2018h-detach: Modifying the LSTM Gradient Towards Better Optimizationcited by 5
  21. 2015BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabulariescited by 3
  22. 2017Language Modeling with Highway LSTMcited by 3
  23. 2020Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Unitscited by 3
  24. 2019Language Modeling through Long Term Memory Networkcited by 2
  25. 2020Language Modelling for Source Code with Transformer-XLcited by 2
  26. 2014Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimationcited by 1
  27. 2019Learning Longer-term Dependencies via Grouped Distributor Unitcited by 1

Built on by

Not cited yet.