mlsanityData to 14 Oct 2026

Jointist: Joint Learning for Multi-instrument Transcription and Its Applications

arXiv 2206.10805 ↗First posted 22 Jun 2022cs.SD cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Jointist, an instrument-aware multi-instrument framework that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip. Jointist consists of the instrument recognition module that conditions the other modules: the transcription module that outputs instrument-specific piano rolls, and the source separation module that utilizes instrument information and transcription results. The instrument conditioning is designed for an explicit multi-instrument functionality while the connection between the transcription and source separation modules is for better transcription performance. Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. However, its novelty necessitates a new perspective on how to evaluate such a model. During the experiment, we assess the model from various aspects, providing a new evaluation perspective for multi-instrument transcription. We also argue that transcription models can be utilized as a preprocessing module for other music analysis tasks. In the experiment on several downstream tasks, the symbolic representation provided by our transcription model turned out to be helpful to spectrograms in solving downbeat detection, chord recognition, and key estimation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 47 of its 79 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

47 of 79 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  8. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  9. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
  10. 2018Music Transformercited by 211
  11. 2017Onsets and Frames: Dual-Objective Piano Transcriptioncited by 94
  12. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
35 more
  1. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  2. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 71
  3. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 64
  4. 2020High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Timescited by 61
  5. 2016Learning Features of Music from Scratchcited by 58
  6. 2019musicnn: Pre-trained convolutional neural networks for music audio taggingcited by 50
  7. 2021Sequence-to-Sequence Piano Transcription with Transformerscited by 48
  8. 2020Evaluation of CNN-based Automatic Music Tagging Modelscited by 41
  9. 2017Transfer learning for music classification and regression taskscited by 37
  10. 2020GiantMIDI-Piano: A large-scale MIDI dataset for classical piano musiccited by 37
  11. 2018MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfercited by 33
  12. 2019Toward Interpretable Music Tagging with Self-Attentioncited by 30
  13. 2017Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveformscited by 27
  14. 2019Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separationscited by 27
  15. 2021Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Datacited by 27
  16. 2021Semi-Supervised Music Tagging Transformercited by 25
  17. 2021ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Datacited by 20
  18. 2019A Bi-directional Transformer for Musical Chord Recognitioncited by 19
  19. 2019Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instrumentscited by 18
  20. 2021A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesiscited by 18
  21. 2022MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Controlcited by 17
  22. 2021SpecTNT: a Time-Frequency Transformer for Music Audiocited by 16
  23. 2019Deep Polyphonic ADSR Piano Note Transcriptioncited by 15
  24. 2019Adversarial Learning for Improved Onsets and Frames Music Transcriptioncited by 13
  25. 2021Omnizart: A General Toolbox for Automatic Music Transcriptioncited by 13
  26. 2017Invariances and Data Augmentation for Supervised Music Transcriptioncited by 12
  27. 2021BERT-like Pre-training for Symbolic Piano Music Classification Taskscited by 12
  28. 2022Modeling Beats and Downbeats with a Time-Frequency Transformercited by 12
  29. 2022To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functionscited by 11
  30. 2020Large-Scale MIDI-based Composer Classificationcited by 10
  31. 2020Deep Composer Classification Using Symbolic Representationcited by 9
  32. 2020Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervisioncited by 9
  33. 2021Revisiting the Onsets and Frames Model with Additive Attentioncited by 9
  34. 2021Music Classification: Beyond Supervised Learning, Towards Real-world Applicationscited by 7
  35. 2020Transformer-XL Based Music Generation with Multiple Sequences of Time-valued Notescited by 5

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2.

6 papers.

  1. 2024Foundation Models for Music: A Surveycited by 37
  2. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
  3. 2022DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capabilitycited by 13
  4. 2023Multitrack Music Transcription with a Time-Frequency Perceivercited by 13
  5. 2024Toward Fully Self-Supervised Multi-Pitch Estimationcited by 5
  6. 2022Show Me the Instruments: Musical Instrument Retrieval from Mixture Audiocited by 4