# CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks

arXiv: https://arxiv.org/abs/2006.02951 · HTML: https://mlsanity.com/paper/2006.02951/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [Large Linguistic Models: Investigating LLMs' metalinguistic abilities](https://mlsanity.com/paper/2305.00948/) (2023)
- [Interpreting intermediate convolutional layers of generative CNNs trained on waveforms](https://mlsanity.com/paper/2104.09489/) (2021)
- [Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networks](https://mlsanity.com/paper/2305.01626/) (2023)
- [Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data](https://mlsanity.com/paper/2203.11476/) (2022)
- [Approaching an unknown communication system by latent space exploration and causal inference](https://mlsanity.com/paper/2303.10931/) (2023)
- [Exploring How Generative Adversarial Networks Learn Phonological Representations](https://mlsanity.com/paper/2305.12501/) (2023)
- [Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales](https://mlsanity.com/paper/2104.08614/) (2021)
- [Interpreting intermediate convolutional layers in unsupervised acoustic word classification](https://mlsanity.com/paper/2110.02375/) (2021)
- [Articulation GAN: Unsupervised modeling of articulatory learning](https://mlsanity.com/paper/2210.15173/) (2022)
- [CiwaGAN: Articulatory information exchange](https://mlsanity.com/paper/2309.07861/) (2023)
- [Unsupervised Learning and Representation of Mandarin Tonal Categories by a Generative CNN](https://mlsanity.com/paper/2509.17859/) (2025)
- [Deciphering Assamese Vowel Harmony with Featural InfoWaveGAN](https://mlsanity.com/paper/2407.06547/) (2024)
- [Big Data and the Computational Social Science of Entrepreneurship and Innovation](https://mlsanity.com/paper/2505.08706/) (2025)
- [A Technique for Isolating Lexically-Independent Phonetic Dependencies in Generative CNNs](https://mlsanity.com/paper/2506.09218/) (2025)
- [Co-Linguistics: AI-augmented Theory Construction in Linguistics](https://mlsanity.com/paper/2609.37635/) (2026)

## Abstract (arXiv)

How can deep neural networks encode information that corresponds to words in human speech into raw acoustic data? This paper proposes two neural network architectures for modeling unsupervised lexical learning from raw acoustic inputs, ciwGAN (Categorical InfoWaveGAN) and fiwGAN (Featural InfoWaveGAN), that combine a Deep Convolutional GAN architecture for audio data (WaveGAN; arXiv:1705.07904) with an information theoretic extension of GAN -- InfoGAN (arXiv:1606.03657), and propose a new latent space structure that can model featural learning simultaneously with a higher level classification and allows for a very low-dimension vector representation of lexical items. Lexical learning is modeled as emergent from an architecture that forces a deep neural network to output data such that unique information is retrievable from its acoustic outputs. The networks trained on lexical items from TIMIT learn to encode unique information corresponding to lexical items in the form of categorical variables in their latent space. By manipulating these variables, the network outputs specific lexical items. The network occasionally outputs innovative lexical items that violate training data, but are linguistically interpretable and highly informative for cognitive modeling and neural network interpretability. Innovative outputs suggest that phonetic and phonological representations learned by the network can be productively recombined and directly paralleled to productivity in human speech: a fiwGAN network trained on `suit' and `dark' outputs innovative `start', even though it never saw `start' or even a [st] sequence in the training data. We also argue that setting latent featural codes to values well beyond training range results in almost categorical generation of prototypical lexical items and reveals underlying values of each latent code.