# Fine-grained Soundscape Control for Augmented Hearing

arXiv: https://arxiv.org/abs/2603.00395 · HTML: https://mlsanity.com/paper/2603.00395/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 74 references that are in the map.
- First posted on arXiv 28 Feb 2026.

## Builds on (most cited first, 29 of 29)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds](https://mlsanity.com/paper/2304.12518/) (2023)
- [Recursive Visual Sound Separation Using Minus-Plus Net](https://mlsanity.com/paper/1908.11602/) (2019)
- [Listen to What You Want: Neural Network-based Universal Sound Selector](https://mlsanity.com/paper/2006.05712/) (2020)
- [Text-Driven Separation of Arbitrary Sounds](https://mlsanity.com/paper/2204.05738/) (2022)
- [FlowSep: Language-Queried Sound Separation with Rectified Flow Matching](https://mlsanity.com/paper/2409.07614/) (2024)
- [One-shot conditional audio filtering of arbitrary sounds](https://mlsanity.com/paper/2011.02421/) (2020)
- [STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency](https://mlsanity.com/paper/2204.09911/) (2022)
- [SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning](https://mlsanity.com/paper/2204.03895/) (2022)
- [DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extraction](https://mlsanity.com/paper/2310.04567/) (2023)
- [Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables](https://mlsanity.com/paper/2311.00320/) (2023)
- [Look Once to Hear: Target Speech Hearing with Noisy Examples](https://mlsanity.com/paper/2405.06289/) (2024)
- [SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer](https://mlsanity.com/paper/2409.08425/) (2024)
- [Don't Separate, Learn to Remix: End-to-End Neural Remixing with Joint Optimization](https://mlsanity.com/paper/2107.13634/) (2021)
- [SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness](https://mlsanity.com/paper/2401.11095/) (2024)
- [Environmental Sound Extraction Using Onomatopoeic Words](https://mlsanity.com/paper/2112.00209/) (2021)
- [Wireless earbuds for low-cost hearing screening](https://mlsanity.com/paper/2212.05435/) (2022)
- [SpaIn-Net: Spatially-Informed Stereophonic Music Source Separation](https://mlsanity.com/paper/2202.07523/) (2022)
- [SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction](https://mlsanity.com/paper/2506.00273/) (2025)
- [Neural Speech Extraction with Human Feedback](https://mlsanity.com/paper/2508.03041/) (2025)
- [OmniBuds: A Sensory Earable Platform for Advanced Bio-Sensing and On-Device Machine Learning](https://mlsanity.com/paper/2410.04775/) (2024)
- [TF-MLPNet: Tiny Real-Time Neural Speech Separation](https://mlsanity.com/paper/2508.03047/) (2025)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)

## Abstract (arXiv)

Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds, and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate real-time performance on 6 ms streaming audio chunks. Across real-world environments in previously unseen indoor and outdoor scenarios, our system enables expressive per-class sound control and achieves substantial improvements in target-class enhancement and interference suppression. Our results show that the world need not be heard as a single, undifferentiated stream: with Aurchestra, the soundscape becomes truly programmable.