mlsanityData to 14 Oct 2026

How Smoothing is N-simplicial Attention?

arXiv 2512.15600 ↗First posted 17 Dec 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Going from pure Multilayer Perceptron (MLP) to a learnable graph message-passing mechanism at each layer has been foundational to state-of-the-art results, despite the computational trade-off (e.g. GATs or Transformers). To go a step further, in this work, we introduce N-simplicial attention, going from pairwise token similarity to higher-order interactions, and adapt it for Rotary Position Embeddings (RoPE). To help manage the increased complexity, we propose a cost-effective simplex selection enabling the model to focus its computation load onto the more task-sensitive interactions. Beyond these core mechanisms, we study how smoothing N-simplicial attention is by deriving a Lipschitz upper-bound and by demonstrating that by itself it also suffers from over-smoothing, despite opening the attention message-passing to higher-order interactions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

47 of 60 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017Graph Attention Networkscited by 8,923
  5. 2016Layer Normalizationcited by 5,669
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  9. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  10. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
35 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  3. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  5. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  6. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 247
  7. 2019Depth-Adaptive Transformercited by 236
  8. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  9. 2020The Lipschitz Constant of Self-Attentioncited by 174
  10. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 166
  11. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  12. 2023The emergence of clusters in self-attention dynamicscited by 140
  13. 2023Representational Strengths and Limitations of Transformerscited by 133
  14. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  15. 2025Why do LLMs attend to the first token?cited by 127
  16. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  17. 2022Not too little, not too much: a theoretical analysis of graph (over)smoothingcited by 106
  18. 2022On the Trade-off between Over-smoothing and Over-squashing in Deep Graph Neural Networkscited by 64
  19. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 53
  20. 2023Demystifying Oversmoothing in Attention-Based Graph Neural Networkscited by 49
  21. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  22. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  23. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  24. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 30
  25. 2025Transformers are Graph Neural Networkscited by 29
  26. 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
  27. 2024The Asymptotic Behavior of Attention in Transformerscited by 16
  28. 2023Unifying over-smoothing and over-squashing in graph neural networks: A physics informed approach and beyondcited by 13
  29. 2024Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformerscited by 12
  30. 2024An end-to-end attention-based approach for learning on graphscited by 11
  31. 2019Logic and the 2-Simplicial Transformercited by 9
  32. 2024On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformerscited by 8
  33. 2025Fast and Simplex: 2-Simplicial Attention in Tritoncited by 8
  34. 2022GEM-2: Next Generation Molecular Property Prediction Network by Modeling Full-range Many-body Interactionscited by 5
  35. 2025Localmax dynamics for attention in transformers and its asymptotic behaviorcited by 2

Built on by

Not cited yet.