mlsanityData to 14 Oct 2026

Word Level Timestamp Generation for Automatic Speech Recognition and Translation

arXiv 2505.15646 ↗First posted 21 May 2025cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model. Accurate timestamp information is crucial for a variety of downstream tasks such as speech content retrieval and timed subtitles. While traditional hybrid systems and end-to-end (E2E) models may employ external modules for timestamp prediction, our approach eliminates the need for separate alignment mechanisms. By leveraging the NeMo Forced Aligner (NFA) as a teacher model, we generate word-level timestamps and train the Canary model to predict timestamps directly. We introduce a new <|timestamp|> token, enabling the Canary model to predict start and end timestamps for each word. Our method demonstrates precision and recall rates between 80% and 90%, with timestamp prediction errors ranging from 20 to 120 ms across four languages, with minimal WER degradation. Additionally, we extend our system to automatic speech translation (AST) tasks, achieving timestamp prediction errors around 200 milliseconds.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

18 of 27 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  8. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  9. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  10. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  11. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  12. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 58
6 more
  1. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  2. 2024CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptionscited by 34
  3. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  4. 2021On Addressing Practical Challenges for RNN-Transducercited by 19
  5. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 15
  6. 2023Unified model for code-switching speech recognition and language identification based on a concatenated tokenizercited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 60
  2. 2026In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictionscited by 1
  3. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1
  4. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  5. 2026Rubato: Transcribing Piano Music with Timestampscited by 0
  6. 2026Relative Time Intervals Representation for Word-level Timestamping with Masked Trainingcited by 0