mlsanityData to 14 Oct 2026

Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale

arXiv 2609.34292 ↗First posted 28 Sep 2026cond-mat.dis-nn cond-mat.mtrl-sci cond-mat.othercited by 0

Not on the map. Why?

Abstract (arXiv)

We report a three-month autonomous research-agent program testing five solid-state-physics-inspired compression mappings on pretrained language models, with predictions committed to git before any pilot data and a 3-sigma gate deciding PASS or SHELVE. The common anchor -- area-law / Kohn-nearsighted decay of the one-particle density matrix -- has a distance face (P001 Wannier, P002 tight-binding) and a rank face (P003 DMRG-truncated MLPs, P005 Wilson-RG, P011 tensor-train embeddings). P005 was pre-empted at Phase 1; three of four Phase-3 pilots were falsified. On the attention face, GPT-2-medium attention-versus-distance is best fit by a stretched exponential in 12 of 16 median-layer heads once probe padding is excluded, and a tight-binding cutoff costs +96% perplexity (P002); on Pythia-160M the Wannier sparsity 0.054 +/- 0.004 is indistinguishable from PCA, random-Haar and identity baselines (P001). On the rank face, per-token tensor-train bond dimension does not track surprisal (r = 0.016 vs a pre-registered 0.65) and the format inflates rather than compresses (P011). P003 is mixed: its scaling claim shelved (r = -0.434), its MPO premise died at stage-0, and its cross-paper check, r = 0.523 as first written, collapses to 0.047 under the same correction, leaving both cross-paper checks null. The results invert the pre-registered prediction that most attention heads behave like Kohn-nearsighted insulators, pointing instead to critical, glassy or heavy-tailed regimes; the inversion is specific to the <= 350M scale tested, while the rank-face no-gain result held to 7-8B. We contribute the pre-registration + 3-sigma + cluster-framing + append-only-catalogue discipline -- including why our own enforcement gate was designed but not deployed -- four pre-registered negative results with full data release, and the inversion. The catalogue holds eighteen concluded studies, seventeen negative.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 66 of its 102 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

66 of 102 references.

References · 66

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2020Rethinking Attention with Performerscited by 1,601
38 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  9. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  10. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  13. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  14. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  15. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  16. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  17. 2022Language model compression with weighted low-rank factorizationcited by 195
  18. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  19. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  20. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  21. 2016Why does deep and cheap learning work so well?cited by 100
  22. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  23. 2020Neural Networks and Quantum Field Theorycited by 61
  24. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 56
  25. 2018Neural Network Renormalization Groupcited by 48
  26. 2016Supervised Learning with Quantum-Inspired Tensor Networkscited by 36
  27. 2017Mutual Information, Neural Networks and the Renormalization Groupcited by 33
  28. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  29. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  30. 2019Compressing deep neural networks by matrix product operatorscited by 28
  31. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  32. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  33. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
  34. 2025Critical attention scaling in long-context transformerscited by 20
  35. 2023TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decompositioncited by 17
  36. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  37. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  38. 2023Pre-registration for Predictive Modelingcited by 13

50 most cited of 66 All 66 (JSON)

Built on by

Not cited yet.