Why voice cloning needs isolated vocals
Voice cloning and AI cover tools build a model of a voice from example recordings. Whatever else is in those recordings gets learned too: a drum hit under every phrase, a guitar ringing behind the vowels, the hiss of a low-quality MP3. The result is a model that sounds metallic or smeared, with a faint backing track stuck to it. Removing the music first leaves the model only the voice to learn from.
FreeVocalRemover separates a mixed recording into two tracks with Hybrid Transformer Demucs, an open-source model trained on hundreds of multitrack songs. Keep the vocal track as training material. If you are making an AI cover, keep the instrumental as well: convert the vocal with your voice model, then mix it back over the music.
Consent comes first
Only clone your own voice, or the voice of someone who has explicitly agreed to it. A cloned voice can put words in a real person's mouth, and the law is catching up: some laws, such as Tennessee's 2024 ELVIS Act, now protect a person's voice from unauthorized AI imitation, and many voice cloning services ask you to confirm that you have the speaker's consent. Label AI covers clearly as AI, and never use a cloned voice to deceive people.
The same goes for the music: only upload songs and recordings you own or have permission to use.
How to prepare training vocals
- Collect your source recordings. Your own songs, stems, demos and voice memos all work. Use the best version of each file you have, ideally a WAV, and favor recordings made with the same microphone.
- Separate each file above. Upload one at a time, up to 6 minutes and 50 MB, and download the vocal track. Recordings that are already a cappella can skip this step.
- Trim and clean. In an editor such as Audacity, cut out long silences, harmonies, crowd noise and any spots where an instrument bled through. A few short, clean clips beat one long, messy one.
- Export in the format your tool expects. Some trainers only accept WAV files, or a specific sample rate. Check the documentation and convert the clips before uploading them.
- Gather enough material. Instant cloning services often need only a minute or two of clean audio. Training your own singing model, for example with RVC, usually works best with at least 10 minutes covering a wide range of notes and vowels.
What makes a good training vocal
- One voice at a time. No harmonies, doubled vocals or backing singers.
- Dry. As little reverb and delay as possible, because the model copies any room or effect it hears.
- Clean. No clipping, distortion, hum or music bleeding through.
- Consistent. The same microphone and a similar level across clips.
- Varied. Low and high notes, soft and loud passages, and plenty of different words.
What AI separation can't fix
Separation removes the instruments, not every problem in a recording. Harmonies and backing vocals stay in the vocal track, because the model treats every voice as one source. Reverb is only partly removed, and heavy effects such as distortion or extreme pitch correction stay baked in. Voice-like instruments, such as saxophone or violin, can leave traces. A low-bitrate source carries its compression artifacts into the vocal, where a voice model will happily learn them. Listen to each clip before adding it to your dataset.
Curious about the model itself? Demucs online lists the exact settings we use, and the vocals extractor covers other ways to use an acapella.