# CiwaGAN: Articulatory information exchange

arXiv: https://arxiv.org/abs/2309.07861 · HTML: https://mlsanity.com/paper/2309.07861/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks](https://mlsanity.com/paper/2006.02951/) (2020)
- [Generative Adversarial Phonology: Modeling unsupervised phonetic and phonological learning with neural networks](https://mlsanity.com/paper/2006.03965/) (2020)
- [Deep Speech Synthesis from MRI-Based Articulatory Representations](https://mlsanity.com/paper/2307.02471/) (2023)
- [Deep Speech Synthesis from Articulatory Representations](https://mlsanity.com/paper/2209.06337/) (2022)
- [Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication](https://mlsanity.com/paper/2009.06110/) (2020)
- [Local and non-local dependency learning and emergence of rule-like representations in speech data by Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/2009.12711/) (2020)
- [EMA2S: An End-to-End Multimodal Articulatory-to-Speech System](https://mlsanity.com/paper/2102.03786/) (2021)
- [Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data](https://mlsanity.com/paper/2203.11476/) (2022)
- [Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAE](https://mlsanity.com/paper/2206.08790/) (2022)
- [Approaching an unknown communication system by latent space exploration and causal inference](https://mlsanity.com/paper/2303.10931/) (2023)
- [Articulation GAN: Unsupervised modeling of articulatory learning](https://mlsanity.com/paper/2210.15173/) (2022)
- [The Mirrornet : Learning Audio Synthesizer Controls Inspired by Sensorimotor Interaction](https://mlsanity.com/paper/2110.05695/) (2021)
- [Learning to Compute the Articulatory Representations of Speech with the MIRRORNET](https://mlsanity.com/paper/2210.16454/) (2022)

## Built on by (most cited first, 1 of 1)

- [Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech](https://mlsanity.com/paper/2409.15525/) (2024)

## Abstract (arXiv)

Humans encode information into sounds by controlling articulators and decode information from sounds using the auditory apparatus. This paper introduces CiwaGAN, a model of human spoken language acquisition that combines unsupervised articulatory modeling with an unsupervised model of information exchange through the auditory modality. While prior research includes unsupervised articulatory modeling and information exchange separately, our model is the first to combine the two components. The paper also proposes an improved articulatory model with more interpretable internal representations. The proposed CiwaGAN model is the most realistic approximation of human spoken language acquisition using deep learning. As such, it is useful for cognitively plausible simulations of the human speech act.