# Labeled Incidence Structures for Native Transformer Modeling of Text, Knowledge Graphs, and Hypergraphs

arXiv: https://arxiv.org/abs/2609.20278 · HTML: https://mlsanity.com/paper/2609.20278/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 25 of 25)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Randomized Positional Encodings Boost Length Generalization of Transformers](https://mlsanity.com/paper/2305.16843/) (2023)
- [Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs](https://mlsanity.com/paper/2003.05425/) (2020)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [On the representation and embedding of knowledge bases beyond binary relations](https://mlsanity.com/paper/1604.08642/) (2016)
- [Contextual Position Encoding: Learning to Count What's Important](https://mlsanity.com/paper/2405.18719/) (2024)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [Knowledge Hypergraph Embedding Meets Relational Algebra](https://mlsanity.com/paper/2102.09557/) (2021)
- [HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position](https://mlsanity.com/paper/2403.19115/) (2024)
- [Hypergraph Transformer for Semi-Supervised Classification](https://mlsanity.com/paper/2312.11385/) (2023)
- [Sentence-Permuted Paragraph Generation](https://mlsanity.com/paper/2104.07228/) (2021)

## Abstract (arXiv)

Current Transformer interfaces index tokens by one or more integer coordinates, which determine their addresses inside attention. In RoPE and its multi-axis or hierarchical variants, the resulting address has the form A(i)=R1i1R2i2R3i3, where the exponents are integer coordinates assigned after choosing a serialized token layout. When Transformers process new or large collections of data, this addressing scheme can produce unseen offsets or coordinate combinations, push repositories toward retrieve-and-serialize pipelines, and force new entities, records, or repository items to be represented by long token strings or identifier embeddings not seen in training. We introduce labeled incidence structures (LIS), in which each participating token or value is an endpoint with content x and a structural index i. The index can include local position, relation role, relation instance, text unit, field, or content-derived identity. The model maps this index to a structural address A(i), so adding new tokens, facts, text units, or repository items applies the same learned address rule to structural and content coordinates rather than requiring larger integer coordinates, unseen coordinate combinations, or new identifier embeddings. Attention scores endpoints i,j using qi^top Pj→ ikj, where journey consistency forces Pj→ i=A(i)-1A(j). When i has several coordinates, such as position, role, and instance, coordinate independence is equivalent to factoring A(i) into one address factor per coordinate. This recovers RoPE, RoPE-2D, and HiRoPE as special cases. This allows knowledge-graph (KG) roles, fact instances, and text units to enter the attention score directly. In controlled shallow diagnostics, the LIS address interface is implemented inside ordinary Transformer attention and yields promising results across text, KG, and n-ary settings.