# Interpreting intermediate convolutional layers in unsupervised acoustic word classification

arXiv: https://arxiv.org/abs/2110.02375 · HTML: https://mlsanity.com/paper/2110.02375/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder](https://mlsanity.com/paper/1603.00982/) (2016)
- [Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems](https://mlsanity.com/paper/1709.04482/) (2017)
- [Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.02720/) (2017)
- [Interpretable Convolutional Filters with SincNet](https://mlsanity.com/paper/1811.09725/) (2018)
- [Towards Visually Grounded Sub-Word Speech Unit Discovery](https://mlsanity.com/paper/1902.08213/) (2019)
- [CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks](https://mlsanity.com/paper/2006.02951/) (2020)
- [Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech](https://mlsanity.com/paper/2103.01032/) (2021)
- [Generative Adversarial Phonology: Modeling unsupervised phonetic and phonological learning with neural networks](https://mlsanity.com/paper/2006.03965/) (2020)
- [Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks](https://mlsanity.com/paper/1304.1018/) (2013)
- [Visualizing Deep Neural Networks for Speech Recognition with Learned Topographic Filter Maps](https://mlsanity.com/paper/1912.04067/) (2019)

## Built on by (most cited first, 3 of 3)

- [Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networks](https://mlsanity.com/paper/2305.01626/) (2023)
- [Approaching an unknown communication system by latent space exploration and causal inference](https://mlsanity.com/paper/2303.10931/) (2023)
- [Unsupervised Learning and Representation of Mandarin Tonal Categories by a Generative CNN](https://mlsanity.com/paper/2509.17859/) (2025)

## Abstract (arXiv)

Understanding how deep convolutional neural networks classify data has been subject to extensive research. This paper proposes a technique to visualize and interpret intermediate layers of unsupervised deep convolutional networks by averaging over individual feature maps in each convolutional layer and inferring underlying distributions of words with non-linear regression techniques. A GAN-based architecture (ciwGAN arXiv:2006.02951) that includes a Generator, a Discriminator, and a classifier was trained on unlabeled sliced lexical items from TIMIT. The training process results in a deep convolutional network that learns to classify words into discrete classes only from the requirement of the Generator to output informative data. This classifier network has no access to the training data -- only to the generated data. We propose a technique to visualize individual convolutional layers in the classifier that yields highly informative time-series data for each convolutional layer and apply it to unobserved test data. Using non-linear regression, we infer underlying distributions for each word which allows us to analyze both absolute values and shapes of individual words at different convolutional layers, as well as perform hypothesis testing on their acoustic properties. The technique also allows us to test individual phone contrasts and how they are represented at each layer.