# Lempel-Ziv Networks

arXiv: https://arxiv.org/abs/2211.13250 · HTML: https://mlsanity.com/paper/2211.13250/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [A comparison of Vector Symbolic Architectures](https://mlsanity.com/paper/2001.11797/) (2020)
- [Learning Longer-term Dependencies in RNNs with Auxiliary Losses](https://mlsanity.com/paper/1803.00144/) (2018)
- [Learning with Holographic Reduced Representations](https://mlsanity.com/paper/2109.02157/) (2021)
- [Research Reproducibility as a Survival Analysis](https://mlsanity.com/paper/2012.09932/) (2020)
- [Shuffling Recurrent Neural Networks](https://mlsanity.com/paper/2007.07324/) (2020)
- [Lempel-Ziv Jaccard Distance, an Effective Alternative to Ssdeep and Sdhash](https://mlsanity.com/paper/1708.03346/) (2017)
- [A Siren Song of Open Source Reproducibility](https://mlsanity.com/paper/2204.04372/) (2022)
- [Deploying Convolutional Networks on Untrusted Platforms Using 2D Holographic Reduced Representations](https://mlsanity.com/paper/2206.05893/) (2022)

## Built on by (most cited first, 3 of 3)

- [Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection](https://mlsanity.com/paper/2403.17978/) (2024)
- [Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations](https://mlsanity.com/paper/2312.15310/) (2023)
- [A Walsh Hadamard Derived Linear Vector Symbolic Architecture](https://mlsanity.com/paper/2410.22669/) (2024)

## Abstract (arXiv)

Sequence processing has long been a central area of machine learning research. Recurrent neural nets have been successful in processing sequences for a number of tasks; however, they are known to be both ineffective and computationally expensive when applied to very long sequences. Compression-based methods have demonstrated more robustness when processing such sequences -- in particular, an approach pairing the Lempel-Ziv Jaccard Distance (LZJD) with the k-Nearest Neighbor algorithm has shown promise on long sequence problems (up to T=200,000,000 steps) involving malware classification. Unfortunately, use of LZJD is limited to discrete domains. To extend the benefits of LZJD to a continuous domain, we investigate the effectiveness of a deep-learning analog of the algorithm, the Lempel-Ziv Network. While we achieve successful proof of concept, we are unable to improve meaningfully on the performance of a standard LSTM across a variety of datasets and sequence processing tasks. In addition to presenting this negative result, our work highlights the problem of sub-par baseline tuning in newer research areas.