mlsanityData to 7 Oct 2026

Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training

arXiv 2302.00286 ↗First posted 1 Feb 2023cs.SD cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Jointist, an instrument-aware multi-instrument framework that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip. Jointist consists of an instrument recognition module that conditions the other two modules: a transcription module that outputs instrument-specific piano rolls, and a source separation module that utilizes instrument information and transcription results. The joint training of the transcription and source separation modules serves to improve the performance of both tasks. The instrument module is optional and can be directly controlled by human users. This makes Jointist a flexible user-controllable framework. Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. Its novelty, however, necessitates a new perspective on how to evaluate such a model. In our experiments, we assess the proposed model from various aspects, providing a new evaluation perspective for multi-instrument transcription. Our subjective listening study shows that Jointist achieves state-of-the-art performance on popular music, outperforming existing multi-instrument transcription models such as MT3. We conducted experiments on several downstream tasks and found that the proposed method improved transcription by more than 1 percentage points (ppt.), source separation by 5 SDR, downbeat detection by 1.8 ppt., chord recognition by 1.4 ppt., and key estimation by 1.4 ppt., when utilizing transcription results obtained from Jointist. Demo available at urlhttps://jointist.github.io/Demo.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 89 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

50 of 89 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  9. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  10. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  11. 2018Music Transformercited by 207
  12. 2017Onsets and Frames: Dual-Objective Piano Transcriptioncited by 93
38 more
  1. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
  2. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  3. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 70
  4. 2020High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Timescited by 61
  5. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  6. 2016Learning Features of Music from Scratchcited by 57
  7. 2019musicnn: Pre-trained convolutional neural networks for music audio taggingcited by 49
  8. 2021Sequence-to-Sequence Piano Transcription with Transformerscited by 48
  9. 2020Evaluation of CNN-based Automatic Music Tagging Modelscited by 39
  10. 2020GiantMIDI-Piano: A large-scale MIDI dataset for classical piano musiccited by 37
  11. 2017Transfer learning for music classification and regression taskscited by 36
  12. 2018MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfercited by 33
  13. 2019Toward Interpretable Music Tagging with Self-Attentioncited by 29
  14. 2017Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveformscited by 27
  15. 2019Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separationscited by 27
  16. 2021Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Datacited by 27
  17. 2021Semi-Supervised Music Tagging Transformercited by 24
  18. 2021ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Datacited by 20
  19. 2019A Bi-directional Transformer for Musical Chord Recognitioncited by 19
  20. 2021A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesiscited by 18
  21. 2019Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instrumentscited by 17
  22. 2022MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Controlcited by 17
  23. 2019Deep Polyphonic ADSR Piano Note Transcriptioncited by 15
  24. 2021SpecTNT: a Time-Frequency Transformer for Music Audiocited by 15
  25. 2019Adversarial Learning for Improved Onsets and Frames Music Transcriptioncited by 13
  26. 2021Omnizart: A General Toolbox for Automatic Music Transcriptioncited by 13
  27. 2017Invariances and Data Augmentation for Supervised Music Transcriptioncited by 12
  28. 2021AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networkscited by 11
  29. 2020Large-Scale MIDI-based Composer Classificationcited by 10
  30. 2022To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functionscited by 10
  31. 2022Modeling Beats and Downbeats with a Time-Frequency Transformercited by 10
  32. 2020Deep Composer Classification Using Symbolic Representationcited by 9
  33. 2020Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervisioncited by 9
  34. 2021Revisiting the Onsets and Frames Model with Additive Attentioncited by 9
  35. 2021Music Classification: Beyond Supervised Learning, Towards Real-world Applicationscited by 7
  36. 2020Transformer-XL Based Music Generation with Multiple Sequences of Time-valued Notescited by 5
  37. 2022A Perceptual Measure for Evaluating the Resynthesis of Automatic Music Transcriptionscited by 4
  38. 2020Toward Expressive Singing Voice Correction: On Perceptual Validity of Evaluation Metrics for Vocal Melody Extractioncited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulationcited by 3
  2. 2024MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakagecited by 2
  3. 2023Self-refining of Pseudo Labels for Music Source Separation with Noisy Labeled Datacited by 1