# Prophecy: Inferring Formal Properties from Neuron Activations

arXiv: https://arxiv.org/abs/2509.21677 · HTML: https://mlsanity.com/paper/2509.21677/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Scenic: A Language for Scenario Specification and Scene Generation](https://mlsanity.com/paper/1809.09310/) (2018)
- [NNrepair: Constraint-based Repair of Neural Network Classifiers](https://mlsanity.com/paper/2103.12535/) (2021)
- [Neural Decision Trees](https://mlsanity.com/paper/1702.07360/) (2017)
- [Probing artificial neural networks: insights from neuroscience](https://mlsanity.com/paper/2104.08197/) (2021)
- [Neural Networks are Decision Trees](https://mlsanity.com/paper/2210.05189/) (2022)
- [Evaluating Deep Neural Networks in Deployment (A Comparative and Replicability Study)](https://mlsanity.com/paper/2407.08730/) (2024)

## Built on by (most cited first, 2 of 2)

- [Feature-Guided Analysis of Neural Networks: A Replication Study](https://mlsanity.com/paper/2511.00052/) (2025)
- [Engineering Resource-constrained Software Systems with DNN Components: a Concept-based Pruning Approach](https://mlsanity.com/paper/2604.09988/) (2026)

## Abstract (arXiv)

We present Prophecy, a tool for automatically inferring formal properties of feed-forward neural networks. Prophecy is based on the observation that a significant part of the logic of feed-forward networks is captured in the activation status of the neurons at inner layers. Prophecy works by extracting rules based on neuron activations (values or on/off statuses) as preconditions that imply certain desirable output property, e.g., the prediction being a certain class. These rules represent network properties captured in the hidden layers that imply the desired output behavior. We present the architecture of the tool, highlight its features and demonstrate its usage on different types of models and output properties. We present an overview of its applications, such as inferring and proving formal explanations of neural networks, compositional verification, run-time monitoring, repair, and others. We also show novel results highlighting its potential in the era of large vision-language models.