# Quantifying Long Range Dependence in Language and User Behavior to improve RNNs

arXiv: https://arxiv.org/abs/1905.09414 · HTML: https://mlsanity.com/paper/1905.09414/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2019.

## Built on by (most cited first, 5 of 5)

- [Dynamic Memory based Attention Network for Sequential Recommendation](https://mlsanity.com/paper/2102.09269/) (2021)
- [Single-Shot Black-Box Adversarial Attacks Against Malware Detectors: A Causal Language Model Approach](https://mlsanity.com/paper/2112.01724/) (2021)
- [Context Uncertainty in Contextual Bandits with Applications to Recommender Systems](https://mlsanity.com/paper/2202.00805/) (2022)
- [DiPS: Differentiable Policy for Sketching in Recommender Systems](https://mlsanity.com/paper/2112.07616/) (2021)
- [Performance Evaluation of Deep Learning Models for Water Quality Index Prediction: A Comparative Study of LSTM, TCN, ANN, and MLP](https://mlsanity.com/paper/2411.01527/) (2024)

## Abstract (arXiv)

Characterizing temporal dependence patterns is a critical step in understanding the statistical properties of sequential data. Long Range Dependence (LRD) --- referring to long-range correlations decaying as a power law rather than exponentially w.r.t. distance --- demands a different set of tools for modeling the underlying dynamics of the sequential data. While it has been widely conjectured that LRD is present in language modeling and sequential recommendation, the amount of LRD in the corresponding sequential datasets has not yet been quantified in a scalable and model-independent manner. We propose a principled estimation procedure of LRD in sequential datasets based on established LRD theory for real-valued time series and apply it to sequences of symbols with million-item-scale dictionaries. In our measurements, the procedure estimates reliably the LRD in the behavior of users as they write Wikipedia articles and as they interact with YouTube. We further show that measuring LRD better informs modeling decisions in particular for RNNs whose ability to capture LRD is still an active area of research. The quantitative measure informs new Evolutive Recurrent Neural Networks (EvolutiveRNNs) designs, leading to state-of-the-art results on language understanding and sequential recommendation tasks at a fraction of the computational cost.