# STAR: Speech-to-Audio Generation via Representation Learning

arXiv: https://arxiv.org/abs/2509.17164 · HTML: https://mlsanity.com/paper/2509.17164/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2025.

## Built on by (most cited first, 1 of 1)

- [AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation](https://mlsanity.com/paper/2610.14697/) (2026)

## Abstract (arXiv)

This work presents STAR, the first end-to-end speech-to-audio generation framework, designed to enhance efficiency and address error propagation inherent in cascaded systems. Unlike prior approaches relying on text or vision, STAR leverages speech as it constitutes a natural modality for interaction. As an initial step to validate the feasibility of the system, we demonstrate through representation learning experiments that spoken sound event semantics can be effectively extracted from raw speech, capturing both auditory events and scene cues. Leveraging the semantic representations, STAR incorporates a bridge network for representation mapping and a two-stage training strategy to achieve end-to-end synthesis. With a 76.9% reduction in speech processing latency, STAR demonstrates superior generation performance over the cascaded systems. Overall, STAR establishes speech as a direct interaction signal for audio generation, thereby bridging representation learning and multimodal synthesis. Generated samples are available at https://zeyuxie29.github.io/STAR.