# MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems

arXiv: https://arxiv.org/abs/2307.11394 · HTML: https://mlsanity.com/paper/2307.11394/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [DiPCo -- Dinner Party Corpus](https://mlsanity.com/paper/1909.13447/) (2019)
- [TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings](https://mlsanity.com/paper/2303.03849/) (2023)
- [Continuous Streaming Multi-Talker ASR with Dual-path Transducers](https://mlsanity.com/paper/2109.08555/) (2021)
- [Multi-turn RNN-T for streaming recognition of multi-party speech](https://mlsanity.com/paper/2112.10200/) (2021)
- [On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems](https://mlsanity.com/paper/2211.16112/) (2022)

## Built on by (most cited first, 28 of 28)

- [NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription](https://mlsanity.com/paper/2401.08887/) (2024)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition](https://mlsanity.com/paper/2310.01688/) (2023)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio](https://mlsanity.com/paper/2505.10975/) (2025)
- [Summary of the NOTSOFAR-1 Challenge: Highlights and Learnings](https://mlsanity.com/paper/2501.17304/) (2025)
- [A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations](https://mlsanity.com/paper/2507.02927/) (2025)
- [The Impact of Automatic Speech Transcription on Speaker Attribution](https://mlsanity.com/paper/2507.08660/) (2025)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition](https://mlsanity.com/paper/2510.03723/) (2025)
- [SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper](https://mlsanity.com/paper/2601.19194/) (2026)
- [TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming](https://mlsanity.com/paper/2608.20958/) (2026)
- [SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge](https://mlsanity.com/paper/2507.03343/) (2025)
- [Error Analysis in a Modular Meeting Transcription System](https://mlsanity.com/paper/2509.10143/) (2025)
- [Target speaker anonymization in multi-speaker recordings](https://mlsanity.com/paper/2510.09307/) (2025)
- [A Unified and Reproducible Experimentation Framework for Speech Understanding](https://mlsanity.com/paper/2605.30899/) (2026)
- [Speaker Tagging Correction With Non-Autoregressive Language Models](https://mlsanity.com/paper/2409.00151/) (2024)
- [Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling](https://mlsanity.com/paper/2508.06393/) (2025)
- [Modeling Overlapped Speech with Shuffles](https://mlsanity.com/paper/2603.17769/) (2026)
- [Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics](https://mlsanity.com/paper/2603.22709/) (2026)
- [An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge](https://mlsanity.com/paper/2607.12468/) (2026)
- [Leading-Silence Augmentation and Multi-Stage Synthetic Supervision for the Second MLC-SLM Challenge](https://mlsanity.com/paper/2608.14150/) (2026)
- [HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding](https://mlsanity.com/paper/2608.29120/) (2026)
- [Transsion's Speaker-Attributed Multilingual ASR System for the MLC-SLM 2026 Challenge](https://mlsanity.com/paper/2609.20833/) (2026)
- [Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition](https://mlsanity.com/paper/2609.30439/) (2026)
- [Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization](https://mlsanity.com/paper/2609.39162/) (2026)
- [Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings](https://mlsanity.com/paper/2609.39344/) (2026)

## Abstract (arXiv)

MeetEval is an open-source toolkit to evaluate all kinds of meeting transcription systems. It provides a unified interface for the computation of commonly used Word Error Rates (WERs), specifically cpWER, ORC-WER and MIMO-WER along other WER definitions. We extend the cpWER computation by a temporal constraint to ensure that only words are identified as correct when the temporal alignment is plausible. This leads to a better quality of the matching of the hypothesis string to the reference string that more closely resembles the actual transcription quality, and a system is penalized if it provides poor time annotations. Since word-level timing information is often not available, we present a way to approximate exact word-level timings from segment-level timings (e.g., a sentence) and show that the approximation leads to a similar WER as a matching with exact word-level annotations. At the same time, the time constraint leads to a speedup of the matching algorithm, which outweighs the additional overhead caused by processing the time stamps.