# GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

arXiv: https://arxiv.org/abs/2502.02942 · HTML: https://mlsanity.com/paper/2502.02942/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2025.

## Built on by (most cited first, 20 of 20)

- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation](https://mlsanity.com/paper/2511.05516/) (2025)
- [Multi-Metric Preference Alignment for Generative Speech Restoration](https://mlsanity.com/paper/2508.17229/) (2025)
- [PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement](https://mlsanity.com/paper/2511.13300/) (2025)
- [FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learning](https://mlsanity.com/paper/2601.16483/) (2026)
- [UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement](https://mlsanity.com/paper/2510.20441/) (2025)
- [Aligning Generative Speech Enhancement with Perceptual Feedback](https://mlsanity.com/paper/2507.09929/) (2025)
- [SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement](https://mlsanity.com/paper/2509.24708/) (2025)
- [MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement](https://mlsanity.com/paper/2507.00966/) (2025)
- [Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR](https://mlsanity.com/paper/2506.13396/) (2025)
- [GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model](https://mlsanity.com/paper/2512.20978/) (2025)
- [Post-Training Speech Enhancement Language Models with Perceptual Rewards](https://mlsanity.com/paper/2606.21458/) (2026)
- [Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English](https://mlsanity.com/paper/2505.17076/) (2025)
- [Absorbing Discrete Diffusion for Speech Enhancement](https://mlsanity.com/paper/2602.22417/) (2026)
- [StuPASE: Towards Low-Hallucination Studio-Quality Generative Speech Enhancement](https://mlsanity.com/paper/2603.09234/) (2026)
- [TokenSE: a Mamba-based discrete token speech enhancement framework for cochlear implants](https://mlsanity.com/paper/2604.12246/) (2026)
- [UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations](https://mlsanity.com/paper/2604.14606/) (2026)
- [SE-AGCNet: An End-to-End Framework for Joint Speech Enhancement and Loudness Control in Meeting Scenarios](https://mlsanity.com/paper/2606.25959/) (2026)
- [Rethinking Language Model-Based Generative Speech Enhancement in the Latent Space of a Neural Audio Codec](https://mlsanity.com/paper/2608.12082/) (2026)
- [Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations](https://mlsanity.com/paper/2609.03940/) (2026)

## Abstract (arXiv)

Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called textitGenSE. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability.