# Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop

arXiv: https://arxiv.org/abs/1904.04063 · HTML: https://mlsanity.com/paper/1904.04063/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 22 of 22)

- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [Evaluating and Steering Modality Preferences in Multimodal Large Language Model](https://mlsanity.com/paper/2505.20977/) (2025)
- [Language Models Use Monotonicity to Assess NPI Licensing](https://mlsanity.com/paper/2105.13818/) (2021)
- [Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries](https://mlsanity.com/paper/2406.10280/) (2024)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [Language model acceptability judgements are not always robust to context](https://mlsanity.com/paper/2212.08979/) (2022)
- [Transformer Circuit Faithfulness Metrics are not Robust](https://mlsanity.com/paper/2407.08734/) (2024)
- [When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs](https://mlsanity.com/paper/2511.02243/) (2025)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL](https://mlsanity.com/paper/2405.06410/) (2024)
- [Phase Transitions in the Output Distribution of Large Language Models](https://mlsanity.com/paper/2405.17088/) (2024)
- [Sparse Attention Decomposition Applied to Circuit Tracing](https://mlsanity.com/paper/2410.00340/) (2024)
- [Discovering Transformer Circuits via a Hybrid Attribution and Pruning Framework](https://mlsanity.com/paper/2510.03282/) (2025)
- [Bored to Death: Artificial Intelligence Research Reveals the Role of Boredom in Suicide Behavior](https://mlsanity.com/paper/2404.14057/) (2024)
- [On Semantic Cognition, Inductive Generalization, and Language Models](https://mlsanity.com/paper/2111.02603/) (2021)
- [A Picture May Be Worth a Thousand Lives: An Interpretable Artificial Intelligence Strategy for Predictions of Suicide Risk from Social Media Images](https://mlsanity.com/paper/2302.09488/) (2023)
- [Controllable Multi-document Summarization: Coverage & Coherence Intuitive Policy with Large Language Model Based Rewards](https://mlsanity.com/paper/2310.03473/) (2023)
- [Episodic Memory Theory for the Mechanistic Interpretation of Recurrent Neural Networks](https://mlsanity.com/paper/2310.02430/) (2023)
- [TinyEmo: Scaling down Emotional Reasoning via Metric Projection](https://mlsanity.com/paper/2410.07062/) (2024)
- [A Mechanistic Understanding of Pronoun Fidelity in LLMs](https://mlsanity.com/paper/2606.16407/) (2026)

## Abstract (arXiv)

The EMNLP 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language. Approaches included: systematic manipulation of input to neural networks and investigating the impact on their performance, testing whether interpretable knowledge can be decoded from intermediate representations acquired by neural networks, proposing modifications to neural network architectures to make their knowledge state or generated output more explainable, and examining the performance of networks on simplified or formal languages. Here we review a number of representative studies in each category.