Given a multi-instrument mixture and query samples, our model, DisMix, extracts and represents individual instruments by a pair of pitch and timbre representations. The two representations act as modular components that can be swapped between instruments to render new audio samples. In the following, we provide random, non cherry-picked, audio samples of the original mixture and the rendered samples by swapping the pitch and timbre representations of instruments between and within mixtures.
Define the reference and target instruments across different mixtures. The reference and target instruments share the same pitch range but have different timbre characteristics.
| mixture | trumpet → flute | violin → viola | viola → saxophone | double bass → bassoon |
|---|---|---|---|---|
| mixture | violin → clarinet | trumpet → flute | cello → saxophone | tuba → tuba |
| mixture | trumpet → oboe | violin → viola | saxophone → trombone | tuba → double bass |
| mixture | oboe → flute | saxophone → trumpet | viola → saxophone | cello → cello |
| mixture | oboe → trumpet | viola → violin | trombone → saxophone | double bass→ tuba |
| mixture | flute → oboe | trumpet → trumpet | saxophone → trombone | cello → double bass |
Define the reference and target instruments within the same mixture. The reference and target instruments could have different pitch ranges.
| mixture | violin → trombone | viola → cello | trombone → viola | cello → violin |
|---|---|---|---|---|
| mixture | oboe → clarinet | clarinet → cello | cello → double bass | double bass → oboe |
| mixture | trumpet → violin | violin → saxophone | saxophone → tuba | tuba → trumpet |
| mixture | oboe → trumpet | trumpet → oboe | trombone → double bass | double bass→ trombone |
| mixture | flute → cello | trumpet → trumpet | saxophone → flute | cello → saxophone |
| mixture | clarinet → saxophone | trumpet → double bass | saxophone → clarinet | double bass→ trumpet |
Exchange sources between different mixtures by swapping individual source-level latents across different mixtures. One part of the SATB parts is separated or isolated from the original mixture and replaced with that part from the other mixture.
| original mixture | isolation (S) | removal (ATB) | replacement (S'ATB) |
|---|---|---|---|
| original mixture | isolation (B) | removal (SAT) | replacement (SATB') |