Music source separation

Starting late 2018, commercial tools became available for the separation of four part stems from a single audio file using AI models. These applications separated stems into Vocal, Drums, Bass and Other from a single audio file. Deep learning advancements and new processing models such as Wav-U-Net for neural networks aided in higher quality and less phase in separations from mixed material. iZotope RX, AudiosourceRe Demix, RipX DeepMix were a handful of early providers of specialized stem separation tools. Eventually this process would become more widely adopted by users and commercial application developers, simultaneously the technologies would continue to improve in terms of quality of separation and speed of separation. The company Deezer also made their Spleeter tool openly available in around 2019 for further research and development into the audio source separation process. Companies such as Apple would go on to release four part stem separation tools directly in the DAW (requiring Apple Silicon in this case), branded as Stem Splitter. There are dozens of companies now utilizing the technologies involved with audio source separation to fit a multitude of applications. New separation options such as piano, strings, winds, guitar, acoustic guitar, and synthesizer are available from a variety of developers as of 2025. The trend for early 2025 was to include these stem separation capabilities with additional AI-based musically purposeful technologies such as mastering, pitch detection and manipulation, chord recognition, lyric transcription, vocal swapping and similar. ==How AI stem separation generally works==

This process involving reverse engineering stems from mastered tracks relies on training models to identify targets in mixtures. Millions of real isolated stems from project files are used to update the parameter margins of models to generate estimates for the final output from mixtures. Large multitrack datasets are developed from the provided isolated stems with further adjustments to mixtures to provide higher numbers in the dataset that train the models for higher degrees of accuracy. Initially providers utilized online-based stem separation because it enable the utilization of powerful computational systems (often involving advanced GPUs), now there are many options for local system based processing of the AI because of optimizations in the processing approach. There are also CPU developments that include neural workflows which facilitate the faster processing architecture needed for highest fidelity stem separation with lower time requirements. ==AI stem separation in sync music==

A growing number of companies are providing the ability for both music publishers and clients to utilize the stem separation technologies for their project needs. Especially useful in the case of vocal removal from mixtures. Utilizing these tools provides editors and agents of the film and TV music industry to quickly have available the ability to adjust and contour songs without the need to reach out to providers which would cause time delays. This improves the potential of a usage because a common issue with sync placements is that certain kinds of sounds can interfere too much with the application of the underscore (for example vocals). This also provides the sync professionals the ability to take the track into unexpected directions and otherwise enhance the mix for the purpose of the application of the track. ==AI stem separation for the DJ==

Quick stem separation is a perfect match for the professional DJ looking to create unique mashups. Generally the track would be rendered into a stem by placing the desired songs into the appropriate folder, when the song is selected it will have the basic four stem groupings available and in some cases individual parts (samples) can be triggered on pads for live performance. ==Notable case studies of AI stem separation==

Disney Music Group used stem separation technologies to enhance their back catalog of recordings. Beatles recordings were split and enhanced with stem separation technologies and the engineers during this process also helped to progress the development of the technology. Numerous classic hit songs have been the target of restoration through stem separation achievements. ==Stems vs AI stem separations==

Stems have been used in the recording industry to mean files bounced during the mixing process, generally a collection of like sounds grouped as a "stem". Stems in the context of the original project files can provide a large number of exported audio files for multiple purposes. These kinds of files generally provide a better quality overall and offer the ability to further isolate project material without introducing artifacts. AI stem separations have generally produced material that is ideally suited for volume adjustments or further effect processing or production. These kinds of stems generally have come in the basic four groupings of vocal, bass, drum, and other. New approaches and deeper training of models resulted in the capability to isolate additional material beyond the basic four groupings, however these kinds of separations generally have spectral anomalies, blend in additional sounds, or change some quality of the original targeted sound. ==Sound design with AI stem separation tools==

The process of using AI and other methodologies to target specific kinds of sounds happened to enable a new method of spectral separation-based sound design through new kinds of tools to edit with such as those in SpectraLayers and RipX. The instant ability to unmix components such as transient information and time-based information into full tracks of unconventional sound creations. Groove shadows and other sound production dubbing techniques are easily achievable by revealing new timbres and structures based on spectral selections because of the advancements into tools to support stem manipulation. ==Noise reduction and AI stem separation==

Aside from advanced noise reduction methods based on learning noise profiles, taking an inverted approach and removing known source targets such as the basic four and specialized models can result in leaving only the noise as a separate track depending on the ensemble. From that, one can remove the noise track. Noise may result on only a single stem and that stem can be targeted exclusively with noise reduction profiles in this way the entire mix does not need to be processed. ==Karaoke (vocal remover) and AI stem separation==

One of the most popular use cases of stem separation is for the purposes of creating an instrumental of a song where one isn't known to exist or available. There are dozens of sites using the technology to attract users aspiring to make such instrumental versions of their favorite songs. ==RipX and the Melodyne approach to stem separation==

The RipX DAW is a unique take on the concept of stem separation because of its note-based harmonic audio visual structure branded as "Rip Audio Format". The system provides a stem separation tool that breaks down a single file into several tracks with notes being represented as the audio track. These notes are highly adjustable and the system includes highly specialized tools for working with the notes and the spectral aspects of the captures. Each note or note part can have specialized effects applied. Tracks can be swapped easily because of the utilization of this notation with other sounds entirely. So the stem is not only separated but the midi is transcribed making it possible to perform as a midi sequence and thereby direct instruments. The notation used by the Rip Audio format resembles the Melodyne architecture of note extraction from audio, these notes however also function as MIDI and audio simultaneously. RipX is a completely unique kind of DAW that is based around stem separation as well as this new Rip Audio format, where audio and midi worlds forge a symbiosis with new kinds of tools to support the new paradigm. ==Stem mastering tool==

Native Instruments created a specialized tool called "Stem Creator Tool" for working with four part stem tracks which is ideally suited for the DJ world as digital DJ consoles and Native Instruments hardware like Traktor and Maschine use the four track stem structure. This tool enables quick mastering and saving of files in a "stem" archival format. The tool is free to use and essential mastering effects applicable to stem-based audio are provided. ==Example approaches and methodologies employed==

Deep learning • Neural networks • Convolutional neural networks (CNNs) • Recurrent neural networks (RNNs) and transformers • Source separation algorithms Signal processing techniques with AI integration • Computational auditory scene analysis (CASA) • Independent component analysis (ICA) • Non-negative matrix factorization (NMF) • Short-time Fourier transform STFT • End-to-end approaches • Hybrid approaches • Masking-based approaches • Repetition-based methods Supporting developments • Ensemble learning • Conv-TasNet • Leveraging large data sets • Mapping-based methods • SynthSOD • Text-based source separation • Wave-U-Net ==Known issues==

The length of time it takes to analyze and separate the sound means that the mixtures generally need pre-rendering or there is a delay in processing. The process of AI-based stem separation produces artifacts and doesn't always result in the correct designation of target instruments. There may also be spectral bleed from part to part. It is easy to compromise the mixed structure of the full work by adjusting certain elements in isolated parts. A rapid volume-modulated sound similar to tremolo may also be a factor of certain kinds of separations. The order in which the source audio is processed and the kind of applications and their sequence can affect the outcome of separations in addition to the kind of mixes and masters. The process can pick up spectral anomalies which may need to be merged into different tracks. There may be a need to reprocess a stem separation of specialized instruments until the desired balance of the captured target sound is realized. Specialized audio editing tools exist to further clean up the processing of the stem separation. == References ==