mlsanityData to 14 Oct 2026

From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification

arXiv 2509.21341 ↗First posted 16 Sep 2025cs.NE cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We study symbolic surrogate modeling of frozen Transformer embeddings to obtain compact, auditable classifiers with calibrated probabilities. For five benchmarks (SST2G, 20NG, MNIST, CIFAR10, MSC17), embeddings from ModernBERT, DINOv2, and SigLIP are partitioned on the training set into disjoint, information-preserving views via semantic-preserving feature partitioning (SPFP). A cooperative multi-population genetic program (MEGP) then learns additive, closed-form logit programs over these views. Across 30 runs per dataset we report F1, AUC, log-loss, Brier, expected calibration error (ECE), and symbolic complexity; a canonical model is chosen by a one-standard-error rule on validation F1 with a parsimony tie-break. Temperature scaling fitted on validation yields substantial ECE reductions on test. The resulting surrogates achieve strong discrimination (up to F1 around 0.99 on MNIST, CIFAR10, MSC17; around 0.95 on SST2G), while 20NG remains most challenging. We provide reliability diagrams, dimension usage and overlap statistics, contribution-based importances, and global effect profiles (PDP and ALE), demonstrating faithful, cross-modal explanations grounded in explicit programs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

27 of 45 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  5. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  6. 2022Transformers in Time Series: A Surveycited by 497
  7. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 404
  8. 2022Multimodal Learning with Transformers: A Surveycited by 400
  9. 2023Explainability for Large Language Models: A Surveycited by 333
  10. 2019Explainable Machine Learning for Scientific Insights and Discoveriescited by 166
  11. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  12. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 125
15 more
  1. 2021SymbolicGPT: A Generative Transformer Model for Symbolic Regressioncited by 105
  2. 2019Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discoverycited by 93
  3. 2019Bayesian Symbolic Regressioncited by 78
  4. 2019Learning Symbolic Physics with Graph Networkscited by 53
  5. 2021Audio Transformerscited by 41
  6. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  7. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 20
  8. 2021Symbolic regression for scientific discovery: an application to wind speed forecastingcited by 16
  9. 2020Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddingscited by 12
  10. 2024Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priorscited by 8
  11. 2023All Roads Lead to Rome? Exploring the Invariance of Transformers' Representationscited by 6
  12. 2025Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representationscited by 5
  13. 2024Semantic-Preserving Feature Partitioning for Multi-View Ensemble Learningcited by 2
  14. 2024A hybrid transformer and attention based recurrent neural network for robust and interpretable sentiment analysis of tweetscited by 2
  15. 2024Graph Neural Networks for Heart Failure Prediction on an EHR-Based Patient Similarity Graphcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Domain-Informed Genetic Superposition Programming: A Case Study on SFRC Beamscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning Transformer Programs
  2. 2026Fully Interpretable Minimal Transformers: From Geometry to Algorithm
  3. 2025BlockCert: Certified Blockwise Extraction of Transformer Mechanisms
  4. 2026Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure
  5. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  6. 2025Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classification
  7. 2025Module-Aware Parameter-Efficient Machine Unlearning on Transformers
  8. 2025TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding