# What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

arXiv: https://arxiv.org/abs/2607.22781 · HTML: https://mlsanity.com/paper/2607.22781/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 24 Jul 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Inductive Representation Learning on Temporal Graphs](https://mlsanity.com/paper/2002.07962/) (2020)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [Towards Better Dynamic Graph Learning: New Architecture and Unified Library](https://mlsanity.com/paper/2303.13047/) (2023)
- [Do We Really Need Complicated Model Architectures For Temporal Networks?](https://mlsanity.com/paper/2302.11636/) (2023)
- [Towards Better Evaluation for Dynamic Link Prediction](https://mlsanity.com/paper/2207.10128/) (2022)
- [Neural Spatio-Temporal Point Processes](https://mlsanity.com/paper/2011.04583/) (2020)
- [Transformer Embeddings of Irregularly Spaced Events and Their Participants](https://mlsanity.com/paper/2201.00044/) (2021)
- [Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks](https://mlsanity.com/paper/2012.08492/) (2020)
- [Neural Point Process for Learning Spatiotemporal Event Dynamics](https://mlsanity.com/paper/2112.06351/) (2021)
- [Normalized Attention Without Probability Cage](https://mlsanity.com/paper/2005.09561/) (2020)
- [Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation](https://mlsanity.com/paper/1907.02204/) (2019)
- [On the Power of Heuristics in Temporal Graphs](https://mlsanity.com/paper/2502.04910/) (2025)
- [Future Link Prediction Without Memory or Aggregation](https://mlsanity.com/paper/2505.19408/) (2025)
- [What Do Temporal Graph Learning Models Learn?](https://mlsanity.com/paper/2510.09416/) (2025)
- [Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention](https://mlsanity.com/paper/2602.23057/) (2026)

## Abstract (arXiv)

High task performance does not show whether a model retains prediction-relevant structural information in its internal representation. Temporal graph models, for example, can achieve high future-link AUC while basic graph statistics remain difficult to recover from the same representation. We identify one source of this gap in the weighted averaging used by standard attention: when an evidence pattern is repeated, the numerator and denominator grow at the same rate, so inputs with different amounts of accumulated evidence can produce the same aggregate. We propose Mass-Aware Attention (MAA), which generalizes standard L1 normalization to an Lp family. Under repetition, MAA makes the numerator and denominator scale at different rates, retaining the effective number of contributing inputs in the representation magnitude. It adds no supervision, parameters, hidden dimensions, or explicit count features, and recovers standard attention at p=1. Across four continuous-time dynamic graph models and three datasets, MAA improves future-link AUC in 11 of 12 model-dataset cells. Linear recovery from the same hidden representation increases by 4.49% on average, and preferential-attachment recovery improves in all 12 cells after family-wise correction. We also observe consistent evidence in marked temporal point processes, temporal knowledge graphs, retrieval-augmented generation, and spatio-temporal point processes. Information accessibility and task utility remain distinct: NLL improves in MTPP, ranking is largely preserved in TKG, additional information in RAG does not improve the diagnostic head, and downstream LayerNorm can erase the signal in STPP. These results position MAA as a general normalization principle for improving predictor-facing representation informativeness by controlling repetition invariance in standard attention.