# MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula

arXiv: https://arxiv.org/abs/2412.15655 · HTML: https://mlsanity.com/paper/2412.15655/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)
- [ASR Error Correction and Domain Adaptation Using Machine Translation](https://mlsanity.com/paper/2003.07692/) (2020)
- [N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space](https://mlsanity.com/paper/2303.00456/) (2023)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Improving Readability for Automatic Speech Recognition Transcription](https://mlsanity.com/paper/2004.04438/) (2020)
- [BART based semantic correction for Mandarin automatic speech recognition system](https://mlsanity.com/paper/2104.05507/) (2021)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)
- [ASR Error Correction with Constrained Decoding on Operation Prediction](https://mlsanity.com/paper/2208.04641/) (2022)
- [Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction](https://mlsanity.com/paper/2407.16370/) (2024)

## Built on by (most cited first, 3 of 3)

- [Intelligibility of Text-to-Speech Systems for Mathematical Expressions](https://mlsanity.com/paper/2506.11086/) (2025)
- [Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysis](https://mlsanity.com/paper/2505.00746/) (2025)
- [Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences](https://mlsanity.com/paper/2508.03542/) (2025)

## Abstract (arXiv)

In various academic and professional settings, such as mathematics lectures or research presentations, it is often necessary to convey mathematical expressions orally. However, reading mathematical expressions aloud without accompanying visuals can significantly hinder comprehension, especially for those who are hearing-impaired or rely on subtitles due to language barriers. For instance, when a presenter reads Euler's Formula, current Automatic Speech Recognition (ASR) models often produce a verbose and error-prone textual description (e.g., e to the power of i x equals cosine of x plus i textitside of x), instead of the concise LaTeX format (i.e.,  eix = cos(x) + isin(x) ), which hampers clear understanding and communication. To address this issue, we introduce MathSpeech, a novel pipeline that integrates ASR models with small Language Models (sLMs) to correct errors in mathematical expressions and accurately convert spoken expressions into structured LaTeX representations. Evaluated on a new dataset derived from lecture recordings, MathSpeech demonstrates LaTeX generation capabilities comparable to leading commercial Large Language Models (LLMs), while leveraging fine-tuned small language models of only 120M parameters. Specifically, in terms of CER, BLEU, and ROUGE scores for LaTeX translation, MathSpeech demonstrated significantly superior capabilities compared to GPT-4o. We observed a decrease in CER from 0.390 to 0.298, and higher ROUGE/BLEU scores compared to GPT-4o.