mlsanityData to 14 Oct 2026

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

arXiv 2608.10288 ↗First posted 10 Aug 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator GLM, built from a positive tensor ALM by elementwise power laws. The architecture is fully specified, verified against pinned reference releases; claims are labeled theorem, conditional theorem, measurement, or conjecture. Unconditionally: PLGA contains SDPA exactly at GLM=I; ALM and AP are strictly entrywise positive, with Perron-Frobenius structure on ALM; the DAG regularizer has the NOTEARS walk-counting form and positivity obstructs exact acyclicity; and, under nonresonance (satisfied by standard rotary frequencies), a commutant criterion identifies which operators preserve relative-position dependence. An inference-collapse theorem: exact input invariance of deductive outputs collapses inference to generalized SDPA with a constant operator. Measured invariance: relative fluctuations of 10-6 and below; perturbation bounds quantify but do not certify cached inference; the assembled proxy misses the decoding margin. A conditional three-stage mechanism (rotary twirl, concentration, row-map contraction) is measured on a released checkpoint. Blockwise training and scoring under the global Gram are stated with explicit target exposure; on tested samples, block and sequential scoring select identical answers and agree on the published TruthfulQA probability-mass metric within 5× 10-5 per item. Self-organized criticality enters as a phenomenological framework with an intrinsic order parameter; open claims become falsifiable conjectures. Selected proof cores are machine-checked in Lean 4.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 51 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

29 of 51 references.

References · 29

  1. 2016Layer Normalizationcited by 5,669
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2016HyperNetworkscited by 1,027
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  10. 2018Bilinear Attention Networkscited by 280
  11. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  12. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 142
17 more
  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  2. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  3. 2020Talking-Heads Attentioncited by 62
  4. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  5. 2024LieRE: Lie Rotational Positional Encodingscited by 18
  6. 2025ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matricescited by 18
  7. 2024Attention as a Hypernetworkcited by 13
  8. 2019CoulGAT: An Experiment on Interpretability of Graph Attention Networkscited by 6
  9. 2021Power Law Graph Transformer for Machine Translation and Representation Learningcited by 5
  10. 2022DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attentioncited by 4
  11. 2024PLDR-LLM: Large Language Model from Power Law Decoder Representationscited by 4
  12. 2025PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inferencecited by 3
  13. 2026Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivitycited by 3
  14. 2026PLDR-LLMs Reason At Self-Organized Criticalitycited by 2
  15. 2026Don't Read Everything: A Curvature-Conditioned Query for Linear Attentioncited by 2
  16. 2026Faster Query-Key Learning Sharpens Attention in Self-Attention Modelscited by 1
  17. 2026Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometrycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.