# BlasBench: An Open Benchmark for Irish Speech Recognition

arXiv: https://arxiv.org/abs/2604.10736 · HTML: https://mlsanity.com/paper/2604.10736/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 12 Apr 2026.

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification](https://mlsanity.com/paper/2402.12654/) (2024)
- [XTREME-S: Evaluating Cross-lingual Speech Representations](https://mlsanity.com/paper/2203.10752/) (2022)
- [Findings of the IWSLT 2024 Evaluation Campaign](https://mlsanity.com/paper/2411.05088/) (2024)
- [Learn and Don't Forget: Adding a New Language to ASR Foundation Models](https://mlsanity.com/paper/2407.06800/) (2024)
- [FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks](https://mlsanity.com/paper/2408.06227/) (2024)
- [What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations](https://mlsanity.com/paper/2409.02449/) (2024)
- [gaBERT -- an Irish Language Model](https://mlsanity.com/paper/2107.12930/) (2021)
- [Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation](https://mlsanity.com/paper/2406.17363/) (2024)
- [A Practitioner's Guide to Building ASR Models for Low-Resource Languages: A Case Study on Scottish Gaelic](https://mlsanity.com/paper/2506.04915/) (2025)
- [How I Built ASR for Endangered Languages with a Spoken Dictionary](https://mlsanity.com/paper/2510.04832/) (2025)

## Abstract (arXiv)

Existing multilingual benchmarks include Irish among dozens of languages but apply no Irish-aware text normalisation, leaving reliable and reproducible ASR comparison impossible. We introduce BlasBench, an open evaluation harness that provides a standalone Irish-aware normaliser preserving fadas, lenition, and eclipsis; a reproducible scoring harness and per-utterance predictions released for all evaluated runs. We pilot this by benchmarking 12 systems across four architecture families on Common Voice ga-IE and FLEURS ga-IE. All Whisper variants exceed 100% WER through insertion-driven hallucination. Microsoft Azure reaches 22.2% WER on Common Voice and 57.5% on FLEURS; the best open model, Omnilingual ASR 7B, reaches 30.65% and 39.09% respectively. Models fine-tuned on Common Voice degrade 33-43 points moving to FLEURS, while massively multilingual models degrade only 7-10 - a generalisation gap that single-dataset evaluation misses.