VORT: Adaptive Power-Law Memory for NLP Transformers

Standard Transformers impose near-exponential decay on the influence of distant tokens, conflicting with the power-law structure of long-range dependencies in natural language. We introduce the Variable-Order Retention Transformer (VORT), a memory architecture in which each ingested token is assigned a learnable fractional order αi∈[δ,1] that governs a Gr"unwald--Letnikov power-law retention kernel. Because the fractional weighted sum is non-Markovian, we approximate it through a sum-of-exponentials (SOE) decomposition computed by Gauss--Laguerre quadrature on a Laplace-type integral representation of the kernel weights. Each exponential component admits a one-step Markovian recurrence at O(Sdv) per step, where S=O(log(T/varepsilon)) terms suffice for varepsilon-uniform accuracy on horizon [1,T]. Retrieval is keyed and associative via a linear-attention accumulator with an exact O(KSdφ dv) -per-step recurrence. Four results are established: (i) an SOE approximation theorem with geometric convergence rate from the analyticity of the integrand after a log-change of variables; (ii) a quantisation bound valid on [δ,1] with correct analysis near α=0; (iii) a direct L2 energy argument (Proposition) showing that for α>1/2 any mixture with fixed minimum decay rate Lambda>0 incurs L2([1,T]) error at least Nα(T)-C(Lambda)→∞, with the Lambda-dependence made explicit; and (iv) linear convergence of a gradient plasticity rule under the Polyak--Lojasiewicz condition. Two synthetic experiments confirm the architectural advantage: a Zipf-distributed retrieval benchmark and an entity label-copy task with uniform lag distribution, the latter ruling out prior-matching as an explanation for the power-law kernel's advantage.