# Closed-Form Interpretation of Neural Network Classifiers with Symbolic Gradients

arXiv: https://arxiv.org/abs/2401.04978 · HTML: https://mlsanity.com/paper/2401.04978/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 51 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [Learning Equations for Extrapolation and Control](https://mlsanity.com/paper/1806.07259/) (2018)
- [Extrapolation and learning equations](https://mlsanity.com/paper/1610.02995/) (2016)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [PySINDy: A comprehensive Python package for robust sparse system identification](https://mlsanity.com/paper/2111.08481/) (2021)
- [AI Poincar'e: Machine Learning Conservation Laws from Trajectories](https://mlsanity.com/paper/2011.04698/) (2020)
- [Rediscovering orbital mechanics with machine learning](https://mlsanity.com/paper/2202.02306/) (2022)
- [Discovering Symmetry Invariants and Conserved Quantities by Interpreting Siamese Neural Networks](https://mlsanity.com/paper/2003.04299/) (2020)
- [Interpretable and Explainable Machine Learning for Materials Science and Chemistry](https://mlsanity.com/paper/2111.01037/) (2021)
- [An Approach to Symbolic Regression Using Feyn](https://mlsanity.com/paper/2104.05417/) (2021)
- [Learning outside the Black-Box: The pursuit of interpretable models](https://mlsanity.com/paper/2011.08596/) (2020)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [OccamNet: A Fast Neural Model for Symbolic Regression at Scale](https://mlsanity.com/paper/2007.10784/) (2020)
- [Correlator Convolutional Neural Networks: An Interpretable Architecture for Image-like Quantum Matter Data](https://mlsanity.com/paper/2011.03474/) (2020)
- [Probing hidden spin order with interpretable machine learning](https://mlsanity.com/paper/1804.08557/) (2018)
- [Symbolic Metamodels for Interpreting Black-boxes Using Primitive Functions](https://mlsanity.com/paper/2302.04791/) (2023)
- [xRAI: Explainable Representations through AI](https://mlsanity.com/paper/2012.06006/) (2020)

## Built on by (most cited first, 3 of 3)

- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [Discovering quantum phenomena with Interpretable Machine Learning](https://mlsanity.com/paper/2604.16015/) (2026)
- [How can we trust opaque systems? Criteria for robust explanations in XAI](https://mlsanity.com/paper/2508.12623/) (2025)

## Abstract (arXiv)

I introduce a unified framework for finding a closed-form interpretation of any single neuron in an artificial neural network. Using this framework I demonstrate how to interpret neural network classifiers to reveal closed-form expressions of the concepts encoded in their decision boundaries. In contrast to neural network-based regression, for classification, it is in general impossible to express the neural network in the form of a symbolic equation even if the neural network itself bases its classification on a quantity that can be written as a closed-form equation. The interpretation framework is based on embedding trained neural networks into an equivalence class of functions that encode the same concept. I interpret these neural networks by finding an intersection between the equivalence class and human-readable equations defined by a symbolic search space. The approach is not limited to classifiers or full neural networks and can be applied to arbitrary neurons in hidden layers or latent spaces.