# Closed-Form Interpretation of Neural Network Latent Spaces with Symbolic Gradients

arXiv: https://arxiv.org/abs/2409.05305 · HTML: https://mlsanity.com/paper/2409.05305/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [Learning Equations for Extrapolation and Control](https://mlsanity.com/paper/1806.07259/) (2018)
- [Discovering physical concepts with neural networks](https://mlsanity.com/paper/1807.10300/) (2018)
- [Extrapolation and learning equations](https://mlsanity.com/paper/1610.02995/) (2016)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [AI Poincar'e: Machine Learning Conservation Laws from Trajectories](https://mlsanity.com/paper/2011.04698/) (2020)
- [Unsupervised learning of phase transitions: from principal component analysis to variational autoencoders](https://mlsanity.com/paper/1703.02435/) (2017)
- [Discovering Symmetry Invariants and Conserved Quantities by Interpreting Siamese Neural Networks](https://mlsanity.com/paper/2003.04299/) (2020)
- [An Approach to Symbolic Regression Using Feyn](https://mlsanity.com/paper/2104.05417/) (2021)
- [Operationally meaningful representations of physical systems in neural networks](https://mlsanity.com/paper/2001.00593/) (2020)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [OccamNet: A Fast Neural Model for Symbolic Regression at Scale](https://mlsanity.com/paper/2007.10784/) (2020)
- [Explainable Representation Learning of Small Quantum States](https://mlsanity.com/paper/2306.05694/) (2023)
- [Machine learning of Kondo physics using variational autoencoders and symbolic regression](https://mlsanity.com/paper/2107.08013/) (2021)
- [A simple framework for contrastive learning phases of matter](https://mlsanity.com/paper/2205.05607/) (2022)

## Built on by (most cited first, 1 of 1)

- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)

## Abstract (arXiv)

It has been demonstrated that artificial neural networks like autoencoders or Siamese networks encode meaningful concepts in their latent spaces. However, there does not exist a comprehensive framework for retrieving this information in a human-readable form without prior knowledge. In quantitative disciplines concepts are typically formulated as equations. Hence, in order to extract these concepts, we introduce a framework for finding closed-form interpretations of neurons in latent spaces of artificial neural networks. The interpretation framework is based on embedding trained neural networks into an equivalence class of functions that encode the same concept. We interpret these neural networks by finding an intersection between the equivalence class and human-readable equations defined by a symbolic search space. Computationally, this framework is based on finding a symbolic expression whose normalized gradients match the normalized gradients of a specific neuron with respect to the input variables. The effectiveness of our approach is demonstrated by retrieving invariants of matrices and conserved quantities of dynamical systems from latent spaces of Siamese neural networks.