mlsanityData to 7 Oct 2026

InstrumentGen: Generating Sample-Based Musical Instruments From Text

arXiv 2311.04339 ↗First posted 7 Nov 2023eess.AS cs.LG cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce the text-to-instrument task, which aims at generating sample-based musical instruments based on textual prompts. Accordingly, we propose InstrumentGen, a model that extends a text-prompted generative audio framework to condition on instrument family, source type, pitch (across an 88-key spectrum), velocity, and a joint text/audio embedding. Furthermore, we present a differentiable loss function to evaluate the intra-instrument timbral consistency of sample-based instruments. Our results establish a foundational text-to-instrument baseline, extending research in the domain of automatic sample-based instrument generation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 22 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

20 of 22 references.

References · 20

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  5. 2022High Fidelity Neural Audio Compressioncited by 673
  6. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2023Simple and Controllable Music Generationcited by 480
  9. 2023MusicLM: Generating Music From Textcited by 442
  10. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  11. 2022AudioGen: Textually Guided Audio Generationcited by 292
  12. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
8 more
  1. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  2. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  3. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 114
  4. 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 72
  5. 2022DDSP-based Singing Vocoders: A New Subtractive-based Synthesizer and A Comprehensive Evaluationcited by 11
  6. 2022GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioningcited by 9
  7. 2020Speech Synthesis and Control Using Differentiable DSPcited by 8
  8. 2022Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfercited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrumentcited by 5
  2. 2025FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Searchcited by 2
  3. 2024Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Modelscited by 1
  4. 2025WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modelingcited by 0