# Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning

arXiv: https://arxiv.org/abs/2601.23169 · HTML: https://mlsanity.com/paper/2601.23169/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Proof Artifact Co-training for Theorem Proving with Language Models](https://mlsanity.com/paper/2102.06203/) (2021)
- [L2-constrained Softmax Loss for Discriminative Face Verification](https://mlsanity.com/paper/1703.09507/) (2017)
- [AdaCos: Adaptively Scaling Cosine Logits for Effectively Learning Deep Face Representations](https://mlsanity.com/paper/1905.00292/) (2019)
- [Teaching Temporal Logics to Neural Networks](https://mlsanity.com/paper/2003.04218/) (2020)
- [The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python](https://mlsanity.com/paper/2305.15507/) (2023)
- [Property Invariant Embedding for Automated Reasoning](https://mlsanity.com/paper/1911.12073/) (2019)
- [From Spot 2.0 to Spot 2.10: What's New?](https://mlsanity.com/paper/2206.11366/) (2022)
- [Permutation Equivariance of Transformers and Its Applications](https://mlsanity.com/paper/2304.07735/) (2023)
- [Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts](https://mlsanity.com/paper/2408.02265/) (2024)
- [Lexical Generalization Improves with Larger Models and Longer Training](https://mlsanity.com/paper/2210.12673/) (2022)
- [Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?](https://mlsanity.com/paper/2505.10443/) (2025)
- [How Does Naming Affect LLMs on Code Analysis Tasks?](https://mlsanity.com/paper/2307.12488/) (2023)
- [When Names Disappear: Revealing What LLMs Actually Understand About Code](https://mlsanity.com/paper/2510.03178/) (2025)
- [Evaluating Structural Generalization in Neural Machine Translation](https://mlsanity.com/paper/2406.13363/) (2024)
- [Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence](https://mlsanity.com/paper/2410.17161/) (2024)

## Built on by (most cited first, 1 of 1)

- [When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification](https://mlsanity.com/paper/2605.07120/) (2026)

## Abstract (arXiv)

Current neural architectures lack a principled way to handle interchangeable tokens, i.e., symbols that are semantically equivalent yet distinguishable, such as bound variables. As a result, models trained on fixed vocabularies often struggle to generalize to unseen symbols, even when the underlying semantics remain unchanged. We propose a novel Transformer-based mechanism that is provably invariant to the renaming of interchangeable tokens. Our approach employs parallel embedding streams to isolate the contribution of each interchangeable token in the input, combined with an aggregated attention mechanism that enables structured information sharing across streams. Experimental results confirm the theoretical guarantees of our method and demonstrate substantial performance gains on open-vocabulary tasks that require generalization to novel symbols. Project page: https://bu-depend-lab.github.io/Symbol-Invariant-Transformer/