# Multiperiodic Processes: Ergodic Sources with a Sublinear Entropy

arXiv: https://arxiv.org/abs/2302.09049 · HTML: https://mlsanity.com/paper/2302.09049/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 81 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 6 of 6)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)

## Built on by (most cited first, 9 of 9)

- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Scaling Laws for Associative Memories](https://mlsanity.com/paper/2310.02984/) (2023)
- [The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets](https://mlsanity.com/paper/2306.14975/) (2023)
- [AI Agents as Universal Task Solvers](https://mlsanity.com/paper/2510.12066/) (2025)
- [LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods](https://mlsanity.com/paper/2407.00322/) (2024)
- [A Language and Its Dimensions: Intrinsic Dimensions of Language Fractal Structures](https://mlsanity.com/paper/2311.10217/) (2023)
- [Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models](https://mlsanity.com/paper/2307.12896/) (2023)
- [The Universal Statistical Structure and Scaling Laws of Chaos and Turbulence](https://mlsanity.com/paper/2311.01358/) (2023)

## Abstract (arXiv)

Several explicit stochastic processes are known to satisfy Hilberg's law, a power-law growth of block entropy conjectured for natural language and recently connected to the neural scaling law. Existing examples either possess a positive Shannon entropy rate, are non-ergodic, or require comparatively involved constructions. We introduce multiperiodic processes, a new class of stationary ergodic processes over the natural numbers generated by random shifts of deterministic multiperiodic sequences. Under mild conditions, multiperiodic processes have vanishing Shannon entropy rate and, under a suitable parameterization, they satisfy both Zipf's law for symbol frequencies and Hilberg's law for block entropy. Since multiperiodic processes are not mixing, we identify the open problem of constructing an elementary strongly mixing source with vanishing entropy rate and Hilberg's law.