# NeuFA: Neural Network Based End-to-End Forced Alignment with Bidirectional Attention Mechanism

arXiv: https://arxiv.org/abs/2203.16838 · HTML: https://mlsanity.com/paper/2203.16838/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2022.

## Built on by (most cited first, 9 of 9)

- [Towards Hierarchical Spoken Language Dysfluency Modeling](https://mlsanity.com/paper/2401.10015/) (2024)
- [Less Peaky and More Accurate CTC Forced Alignment by Label Priors](https://mlsanity.com/paper/2406.02560/) (2024)
- [Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model](https://mlsanity.com/paper/2301.12343/) (2023)
- [Speech-text based multi-modal training with bidirectional attention for improved speech recognition](https://mlsanity.com/paper/2211.00325/) (2022)
- [Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video](https://mlsanity.com/paper/2303.08670/) (2023)
- [Cross-Attention is Half Explanation in Speech-to-Text Models](https://mlsanity.com/paper/2509.18010/) (2025)
- [Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learning](https://mlsanity.com/paper/2503.03286/) (2025)
- [FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech](https://mlsanity.com/paper/2609.32396/) (2026)
- [Where Words Begin and End](https://mlsanity.com/paper/2610.17782/) (2026)

## Abstract (arXiv)

Although deep learning and end-to-end models have been widely used and shown superiority in automatic speech recognition (ASR) and text-to-speech (TTS) synthesis, state-of-the-art forced alignment (FA) models are still based on hidden Markov model (HMM). HMM has limited view of contextual information and is developed with long pipelines, leading to error accumulation and unsatisfactory performance. Inspired by the capability of attention mechanism in capturing long term contextual information and learning alignments in ASR and TTS, we propose a neural network based end-to-end forced aligner called NeuFA, in which a novel bidirectional attention mechanism plays an essential role. NeuFA integrates the alignment learning of both ASR and TTS tasks in a unified framework by learning bidirectional alignment information from a shared attention matrix in the proposed bidirectional attention mechanism. Alignments are extracted from the learnt attention weights and optimized by the ASR, TTS and FA tasks in a multi-task learning manner. Experimental results demonstrate the effectiveness of our proposed model, with mean absolute error on test set drops from 25.8 ms to 23.7 ms at word level, and from 17.0 ms to 15.7 ms at phoneme level compared with state-of-the-art HMM based model.