# Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure

arXiv: https://arxiv.org/abs/2609.23551 · HTML: https://mlsanity.com/paper/2609.23551/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Self-Attention with Structural Position Representations](https://mlsanity.com/paper/1909.00383/) (2019)
- [LieRE: Lie Rotational Positional Encodings](https://mlsanity.com/paper/2406.10322/) (2024)
- [HDT: Hierarchical Document Transformer](https://mlsanity.com/paper/2407.08330/) (2024)
- [Sentence-Permuted Paragraph Generation](https://mlsanity.com/paper/2104.07228/) (2021)

## Abstract (arXiv)

Standard positional encodings treat position as a one-dimensional reading-order coordinate, but reading order alone does not determine hierarchical textual structure. We use a hierarchical rotary positional encoding (hRoPE) that represents paragraph, sentence, and token indices as separate channels, hold the token sequence fixed, intervene on the paragraph coordinate p1, and measure cross-paragraph attention with a token-distance-exact estimator. Attention is compressed in every corpus, but compression alone is not diagnostic of true structure: an architecturally identical channel with density-matched random labels is compressed too. What distinguishes real structure is the depth of compression: the real-versus-random depth gap is resolvable in two of three corpora and not in the third, and real-structure depth varies more strongly across corpora than the control's. Comparing eight corpus-only quantities across three constructs (lexical persistence, paragraph length, embedding-based coherence), none fully reproduces the cross-corpus ordering of depth, though embedding-based coherence comes closest. At the paragraph level that relation transfers as a common slope, but with the opposite sign to the corpus-level ranking: controlling for length, diversity, and position, paragraphs with more similar neighbors compress less deeply, with no detectable slope difference between any pair of corpora, while length, lexical-diversity, and position effects remain corpus-specific. Compression depth, not its location, is the reproducible signature of genuine paragraph structure in our setting.