# A Linear Dynamical System Model for Text

arXiv: https://arxiv.org/abs/1502.04081 · HTML: https://mlsanity.com/paper/1502.04081/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2015.

## Built on by (most cited first, 6 of 6)

- [Auto-Regressive Next-Token Predictors are Universal Learners](https://mlsanity.com/paper/2309.06979/) (2023)
- [Model-free Online Learning for the Kalman Filter: Forgetting Factor and Logarithmic Regret](https://mlsanity.com/paper/2505.08982/) (2025)
- [The Complexity of Sequential Prediction in Dynamical Systems](https://mlsanity.com/paper/2402.06614/) (2024)
- [Linear RNNs Provably Learn Linear Dynamic Systems](https://mlsanity.com/paper/2211.10582/) (2022)
- [Spectral learning of Bernoulli linear dynamical systems models](https://mlsanity.com/paper/2303.02060/) (2023)
- [Nonlinear Kalman Filtering with Reparametrization Gradients](https://mlsanity.com/paper/2303.04450/) (2023)

## Abstract (arXiv)

Low dimensional representations of words allow accurate NLP models to be trained on limited annotated data. While most representations ignore words' local context, a natural way to induce context-dependent representations is to perform inference in a probabilistic latent-variable sequence model. Given the recent success of continuous vector space word representations, we provide such an inference procedure for continuous states, where words' representations are given by the posterior mean of a linear dynamical system. Here, efficient inference can be performed using Kalman filtering. Our learning algorithm is extremely scalable, operating on simple cooccurrence counts for both parameter initialization using the method of moments and subsequent iterations of EM. In our experiments, we employ our inferred word embeddings as features in standard tagging tasks, obtaining significant accuracy improvements. Finally, the Kalman filter updates can be seen as a linear recurrent neural network. We demonstrate that using the parameters of our model to initialize a non-linear recurrent neural network language model reduces its training time by a day and yields lower perplexity.