mlsanityData to 14 Oct 2026

Free Energy Mixer

arXiv 2602.07160 ↗First posted 6 Feb 2026cs.CL cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Standard attention stores keys/values losslessly but reads them via a per-head convex average, blocking channel-wise selection. We propose the Free Energy Mixer (FEM): a free-energy (log-sum-exp) read that applies a value-driven, per-channel log-linear tilt to a fast prior (e.g., from queries/keys in standard attention) over indices. Unlike methods that attempt to improve and enrich the (q,k) scoring distribution, FEM treats it as a prior and yields a value-aware posterior read at unchanged complexity, smoothly moving from averaging to per-channel selection as the learnable inverse temperature increases, while still preserving parallelism and the original asymptotic complexity (O(T2) for softmax; O(T) for linearizable variants). We instantiate a two-level gated FEM that is plug-and-play with standard and linear attention, linear RNNs and SSMs. It consistently outperforms strong baselines on NLP, vision, and time-series at matched parameter budgets.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 65 of its 77 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

65 of 77 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  4. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  13. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  14. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  15. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  16. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  17. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  18. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  19. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  20. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  21. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  22. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  23. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  24. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  25. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  26. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  27. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  28. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  29. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  30. 2023Larger language models do in-context learning differentlycited by 394
  31. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  32. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  33. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  34. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  35. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  36. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  37. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  38. 2022Transformer Quality in Linear Timecited by 191

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CAST: Causal Anchored Simplex Transport for Distribution-Valued Time Seriescited by 0
  2. 2026Target-Token Asymmetry Can Masquerade As Depth Specializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SANTA++: Sampling Attention through Representative Keys
  2. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  3. 2026MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation
  4. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  5. 2026COBS: Cumulant Order Block Sparse Attention
  6. 2026Depth-Attention: Cross-Layer Value Mixing for Language Models
  7. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
  8. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs