Abstract
Music source separation (MSS) aims to extract individual instrument sources from their mixture. While most existing methods focus on the widely adopted four-stem separation setup (vocals, bass, drums, and other instruments), this approach lacks the flexibility needed for real-world applications. To address this, we propose GuideSep, a diffusion-based MSS model capable of instrument-agnostic separation beyond the four-stem setup. GuideSep is conditioned on multiple inputs: a waveform mimicry condition, which can be easily provided by humming or playing the target melody, and mel-spectrogram domain masks, which offer additional guidance for separation. Unlike prior approaches that relied on fixed class labels or sound queries, our conditioning scheme, coupled with the generative approach, provides greater flexibility and applicability. Additionally, we design a mask-prediction baseline using the same model architecture to systematically compare predictive and generative approaches. Our objective and subjective evaluations demonstrate that GuideSep achieves high-quality separation while enabling more versatile instrument extraction, highlighting the potential of user participation in the diffusion-based generative process for MSS.
Separation Result from Real-world Examples
This section we show the separation results of GuideSep on the real-world music. We condition the model on our recorded humming or virtual instrument. The ground-truth is not provided as there's no access to the ground-truth of the real-world music. The user-input masks are shown in the last column. Masks are shown overlay on top of the mixture Mel-spectrogram.
Condition Type
Mixture
Mimicry Condition
GuideSep Result
Mel-mask from User
Mimicry + Pseudo-Mel-mask
Mimicry
Mimicry + Mel-mask
Mimicry
Mimicry + Mel-mask
Mel-mask
Mimicry
Mimicry
Mimicry + Mel-mask
Evaluation Samples from Slakh2100 Test Split
Conditioned on Humming Only
This section we show the separation results of GuideSep on the Slakh2100 test split. We condition the model on our recorded humming signal only. The humming signal is used as the mimicry condition to guide the separation process.
Target Instrument
Mixture
Mimicry Condition
GuideSep Result
Ground-truth
Piano
Guitar
Strings
Bass
Brass
Synth
Pipe
Reed
Organ
Chromatic Percussion
From the brass example above, we hear that the brass and electric guitar share the same melody, and the separated result contains both electric guitar and brass. Here, we demonstrate how our prior knowledge about the target instrument—in this mixture the brass contains more high-frequency harmonics compared to electric guitar—can be combined with Mel-spectrogram masks to separate the brass from the mixture. Below, we present the separation results for the same mixture using both the mimicry condition and the Mel-spectrogram mask. The separation result is now much cleaner.
Target Instrument
Mixture
Mimicry Condition
GuideSep Result w/out Mask
GuideSep Result w/ Mask
Ground-truth
Brass
Mixture Mel-spectrogram
Mixture Mel-spectrogram with user input mask overlay
Conditioned on Simulated Mimicry Condition Signal and Mel-masks
Here we show the separation result of GuideSep on the Slakh2100 test split. We condition the model on the simulated mimicry condition signal and Mel-spectrogram masks which corresponds to our evaluation setup in the paper.
