mlsanityData to 14 Oct 2026

SingSong: Generating musical accompaniments from singing

arXiv 2301.12662 ↗First posted 30 Jan 2023cs.SD cs.AI cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be suitable for conditional "audio-to-audio" generation tasks, and train it on the source-separated (vocal, instrumental) pairs. In a pairwise comparison with the same vocal inputs, listeners expressed a significant preference for instrumentals generated by SingSong compared to those from a strong retrieval baseline. Sound examples at https://g.co/magenta/singsong

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

29 of 38 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  7. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2020Jukebox: A Generative Model for Musiccited by 542
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  11. 2018Adversarial Audio Synthesiscited by 225
  12. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
17 more
  1. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  2. 2020DDSP: Differentiable Digital Signal Processingcited by 178
  3. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  4. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 116
  5. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  6. 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 73
  7. 2016madmom: a new Python Audio and Music Signal Processing Librarycited by 64
  8. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  9. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  10. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  11. 2021KUIELab-MDX-Net: A Two-Stream Neural Network for Music Demixingcited by 24
  12. 2020Automatic Melody Harmonization with Triad Chords: A Comparative Studycited by 19
  13. 2019High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interactioncited by 14
  14. 2022JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAEcited by 11
  15. 2021vocadito: A dataset of solo vocals with f0, note, and lyric annotationscited by 8
  16. 2018Genre-Agnostic Key Classification With Convolutional Neural Networkscited by 2
  17. 2022Redefining Relationships in Musiccited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 24; 2025: 12; 2026: 6.

55 papers.

Sort:

By area:

  1. 2023Simple and Controllable Music Generationcited by 488
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  4. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  5. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  6. 2023Textually Pretrained Speech Language Modelscited by 101
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  8. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  9. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 68
  10. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  11. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  12. 2024ComposerX: Multi-Agent Symbolic Music Composition with LLMscited by 42
38 more
  1. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 40
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2023Anticipatory Music Transformercited by 36
  4. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 33
  5. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  6. 2023StemGen: A music generation model that listenscited by 32
  7. 2024GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Taskscited by 30
  8. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
  9. 2023Content-based Controls For Music Large Language Modelingcited by 19
  10. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  11. 2023LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPTcited by 15
  12. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  13. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Modelscited by 13
  14. 2024Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompanimentcited by 13
  15. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  16. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  17. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  18. 2023Brain2Music: Reconstructing Music from Human Brain Activitycited by 10
  19. 2025Live Music Modelscited by 10
  20. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 9
  21. 2023TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognitioncited by 8
  22. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  23. 2025ReaLJam: Real-Time Human-AI Music Jamming with Reinforcement Learning-Tuned Transformerscited by 7
  24. 2024FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generationcited by 5
  25. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  26. 2024Multi-Source Music Generation with Latent Diffusioncited by 4
  27. 2025Hookpad Aria: A Copilot for Songwriterscited by 4
  28. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  29. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  30. 2024Sound Check: Auditing Audio Datasetscited by 3
  31. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  32. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  33. 2025Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notationcited by 3
  34. 2025AnyAccomp: Generalizable Accompaniment Generation via Quantized Melodic Bottleneckcited by 2
  35. 2026DARC: Drum accompaniment generation with fine-grained rhythm controlcited by 2
  36. 2026Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSPcited by 2
  37. 2025WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modelingcited by 0
  38. 2025SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusioncited by 0

50 most cited of 55 All 55 (JSON)