mlsanityData to 7 Oct 2026

Explicit Word Density Estimation for Language Modelling

arXiv 2406.10256 ↗First posted 10 Jun 2024cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Language Modelling has been a central part of Natural Language Processing for a very long time and in the past few years LSTM-based language models have been the go-to method for commercial language modeling. Recently, it has been shown that when looking at language modelling from a matrix factorization point of view, the final Softmax layer limits the expressiveness of the model, by putting an upper bound on the rank of the resulting matrix. Additionally, a new family of neural networks based called NeuralODEs, has been introduced as a continuous alternative to Residual Networks. Moreover, it has been shown that there is a connection between these models and Normalizing Flows. In this work we propose a new family of language models based on NeuralODEs and the continuous analogue of Normalizing Flows and manage to improve on some of the baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 36 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

15 of 36 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Variational Inference with Normalizing Flowscited by 2,161
  3. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  4. 2016HyperNetworkscited by 1,003
  5. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  6. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  7. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  8. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  9. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  10. 2016Exploring the Limits of Language Modelingcited by 172
  11. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  12. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
3 more
  1. 2018Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputscited by 26
  2. 2019Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearitiescited by 12
  3. 2018Direct Output Connection for a High-Rank Language Modelcited by 10

Built on by

Not cited yet.