# When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification

arXiv: https://arxiv.org/abs/2605.07120 · HTML: https://mlsanity.com/paper/2605.07120/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 68 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 4 of 4)

- [How Do Transformers Learn Variable Binding in Symbolic Programs?](https://mlsanity.com/paper/2505.20896/) (2025)
- [In-Context Algebra](https://mlsanity.com/paper/2512.16902/) (2025)
- [Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning](https://mlsanity.com/paper/2601.23169/) (2026)
- [To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning](https://mlsanity.com/paper/2604.21632/) (2026)

## Abstract (arXiv)

Template tasks have emerged as a clean testbed for asking whether transformers reason with abstract symbols rather than concrete token names. We study the fixed-label classification version of this problem, where train and test examples share latent templates but may use disjoint vocabularies. Unlike next-token prediction, the model need not emit unseen symbols; it must learn a decision rule invariant to symbol renaming. We analyze regularized kernel logistic classification in the transformer-kernel regime. Our main result decomposes the learned predictor into an ideal template-level classifier and a finite-sample perturbation caused by accidental token overlaps in the training data. We encode these overlaps by a colored collision graph and prove high-probability margin-transfer guarantees for fresh-symbol classification. This perspective extends template-based analyses to logistic classification and refines scalar diversity conditions: vocabulary size controls the average rate of collisions, but collision geometry controls whether the ideal classification margin is preserved. More broadly, the same perturbation framework applies to abstraction-augmented inputs, yielding a general margin-versus-collision criterion for identifying when prompting strategies improve fresh-symbol generalization. Synthetic template experiments illustrate the predicted roles of regularization, sample size, and transformer-kernel structure.