# Normalizing Flow based Hidden Markov Models for Classification of Speech Phones with Explainability

arXiv: https://arxiv.org/abs/2107.00730 · HTML: https://mlsanity.com/paper/2107.00730/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 1 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Light Gated Recurrent Units for Speech Recognition](https://mlsanity.com/paper/1803.10225/) (2018)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [Normalizing Flows: An Introduction and Review of Current Methods](https://mlsanity.com/paper/1908.09257/) (2019)
- [Diverse feature visualizations reveal invariances in early layers of deep neural networks](https://mlsanity.com/paper/1807.10589/) (2018)
- [Segmental Recurrent Neural Networks for End-to-end Speech Recognition](https://mlsanity.com/paper/1603.00223/) (2016)
- [Powering Hidden Markov Model by Neural Network based Generative Models](https://mlsanity.com/paper/1910.05744/) (2019)
- [Robust Classification using Hidden Markov Models and Mixtures of Normalizing Flows](https://mlsanity.com/paper/2102.07284/) (2021)
- [Neural Network based Explicit Mixture Models and Expectation-maximization based Learning](https://mlsanity.com/paper/1907.13432/) (2019)

## Built on by (most cited first, 4 of 4)

- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [Neural HMMs are all you need (for high-quality attention-free TTS)](https://mlsanity.com/paper/2108.13320/) (2021)
- [OverFlow: Putting flows on top of neural transducers for better TTS](https://mlsanity.com/paper/2211.06892/) (2022)
- [A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing](https://mlsanity.com/paper/2607.16183/) (2026)

## Abstract (arXiv)

In pursuit of explainability, we develop generative models for sequential data. The proposed models provide state-of-the-art classification results and robust performance for speech phone classification. We combine modern neural networks (normalizing flows) and traditional generative models (hidden Markov models - HMMs). Normalizing flow-based mixture models (NMMs) are used to model the conditional probability distribution given the hidden state in the HMMs. Model parameters are learned through judicious combinations of time-tested Bayesian learning methods and contemporary neural network learning methods. We mainly combine expectation-maximization (EM) and mini-batch gradient descent. The proposed generative models can compute likelihood of a data and hence directly suitable for maximum-likelihood (ML) classification approach. Due to structural flexibility of HMMs, we can use different normalizing flow models. This leads to different types of HMMs providing diversity in data modeling capacity. The diversity provides an opportunity for easy decision fusion from different models. For a standard speech phone classification setup involving 39 phones (classes) and the TIMIT dataset, we show that the use of standard features called mel-frequency-cepstral-coeffcients (MFCCs), the proposed generative models, and the decision fusion together can achieve 86.6% accuracy by generative training only. This result is close to state-of-the-art results, for examples, 86.2% accuracy of PyTorch-Kaldi toolkit [1], and 85.1% accuracy using light gated recurrent units [2]. We do not use any discriminative learning approach and related sophisticated features in this article.