Multi-Source Diffusion Models for Simultaneous Music Generation and Separation

Abstract

In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e. generating a mixture, separating the sources), we also introduce and experiment on the partial inference task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.

Publication
arXiv
Giorgio Mariani
Giorgio Mariani
PhD Student

Refactor specialist

Irene Tallini
Irene Tallini
PhD Student

I’m a Computer Science PhD with Math Bachelor and passion. Right now I’m working on AI for Music. I like to sing, also.

Emilian Postolache
Emilian Postolache
PhD Student

Generative Audio Maverick

Michele Mancusi
Michele Mancusi
PhD Student

PhD Student @SapienzaRoma CS | Intern @Musixmatch

Luca Cosmo
Luca Cosmo
Assistant Professor
Emanuele Rodolà
Emanuele Rodolà
Full Professor