mlsanityData to 14 Oct 2026

Hierarchical vs. Flat Iteration in Shared-Weight Transformers

arXiv 2604.14442 ↗First posted 15 Apr 2026cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

We present an empirical study of whether hierarchically structured, shared-weight recurrence can match the representational quality of independent-layer stacking in a Transformer-based language model. HRM-LM replaces L independent Transformer layers with a two-speed recurrent pair: a Fast module operating at every step for local refinement, and a Slow module operating every T steps for global compression. This recurrent hierarchy is unrolled for M = N x T steps with shared parameters. The central and most robust finding, supported by a parameter-matched Universal Transformer ablation (UniTF, 1.2B) across five independent runs, is a sharp empirical gap between the two approaches.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

25 of 35 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
13 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2018Universal Transformerscited by 647
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  6. 2019Deep Equilibrium Modelscited by 615
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  8. 2025Hierarchical Reasoning Modelcited by 177
  9. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  10. 2023White-Box Transformers via Sparse Rate Reductioncited by 80
  11. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  12. 2025PolarQuant: Quantizing KV Caches with Polar Transformationcited by 27
  13. 2024Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Searchcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  2. 2026Loop the Loopies!cited by 8
  3. 2026CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stabilitycited by 4
  4. 2026LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scalingcited by 4