# A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer

arXiv: https://arxiv.org/abs/2608.20394 · HTML: https://mlsanity.com/paper/2608.20394/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 11 of 11)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models](https://mlsanity.com/paper/2402.13064/) (2024)
- [COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning](https://mlsanity.com/paper/2311.02248/) (2023)
- [SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning](https://mlsanity.com/paper/2504.09081/) (2025)
- [From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation](https://mlsanity.com/paper/2507.08924/) (2025)

## Abstract (arXiv)

Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage's marginal value unknown. We design a production-ready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2x2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.4B-70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and 3 downstream MCQA benchmarks. Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains. 4-judge quality rises consistently, yet the cross-model mean MCQA gain is not significant; positive transfer concentrates on family-domain aligned pairs. This differential pattern is consistent with a format mismatch: Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall. All six human raters report higher full-pipeline quality, confirming the LLM-judge direction. An STT-engine swap to Whisper-medium confirms pipeline robustness. A non-hallucination audit shows the two frontier LLMs admit unknown on approximately 8% of QA on average; we release samples, prompts, code, and all SFT checkpoints.