mlsanityData to 7 Oct 2026

What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

arXiv 2607.22781 ↗First posted 24 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

High task performance does not show whether a model retains prediction-relevant structural information in its internal representation. Temporal graph models, for example, can achieve high future-link AUC while basic graph statistics remain difficult to recover from the same representation. We identify one source of this gap in the weighted averaging used by standard attention: when an evidence pattern is repeated, the numerator and denominator grow at the same rate, so inputs with different amounts of accumulated evidence can produce the same aggregate. We propose Mass-Aware Attention (MAA), which generalizes standard L1 normalization to an Lp family. Under repetition, MAA makes the numerator and denominator scale at different rates, retaining the effective number of contributing inputs in the representation magnitude. It adds no supervision, parameters, hidden dimensions, or explicit count features, and recovers standard attention at p=1. Across four continuous-time dynamic graph models and three datasets, MAA improves future-link AUC in 11 of 12 model-dataset cells. Linear recovery from the same hidden representation increases by 4.49% on average, and preferential-attachment recovery improves in all 12 cells after family-wise correction. We also observe consistent evidence in marked temporal point processes, temporal knowledge graphs, retrieval-augmented generation, and spatio-temporal point processes. Information accessibility and task utility remain distinct: NLL improves in MTPP, ranking is largely preserved in TKG, additional information in RAG does not improve the diagnostic head, and downstream LayerNorm can erase the signal in STPP. These results position MAA as a general normalization principle for improving predictor-facing representation informativeness by controlling repetition invariance in standard attention.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

24 of 37 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2019Designing and Interpreting Probes with Control Taskscited by 608
  8. 2020Inductive Representation Learning on Temporal Graphscited by 454
  9. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  10. 2019Predicting Dynamic Embedding Trajectory in Temporal Interaction Networkscited by 443
  11. 2020Transformer Hawkes Processcited by 230
  12. 2023Towards Better Dynamic Graph Learning: New Architecture and Unified Librarycited by 154
12 more
  1. 2023Do We Really Need Complicated Model Architectures For Temporal Networks?cited by 141
  2. 2022Towards Better Evaluation for Dynamic Link Predictioncited by 120
  3. 2020Neural Spatio-Temporal Point Processescited by 92
  4. 2021Transformer Embeddings of Irregularly Spaced Events and Their Participantscited by 90
  5. 2020Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networkscited by 84
  6. 2021Neural Point Process for Learning Spatiotemporal Event Dynamicscited by 42
  7. 2020Normalized Attention Without Probability Cagecited by 19
  8. 2019Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservationcited by 17
  9. 2025On the Power of Heuristics in Temporal Graphscited by 11
  10. 2025Future Link Prediction Without Memory or Aggregationcited by 5
  11. 2025What Do Temporal Graph Learning Models Learn?cited by 4
  12. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Eigenvalues as a Metric for Memory Dynamics in Sequence Models
  2. 2026Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
  3. 2026SANTA++: Sampling Attention through Representative Keys
  4. 2026A Comparative Analysis of Attention versus State-Space Models for In-Context Learning
  5. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
  6. 2026Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction
  7. 2026Geometry-Aware Attention Guidance for Diffusion Models via Modern Hopfield Dynamics
  8. 2026State Rank Dynamics in Linear Attention LLMs