iframe<table
How Does Singing Voice Conversion Work? The Seed-VC Baseline
Have you ever tried singing? What about extreme rock vocal techniques, like hoarse vocals? What about doing it the same way across time? This singing technique is not as easy as it seems and requires consistent skills especially if you do it long-term. Sometimes the voice changes but the singing style should remain the same.<table
Vocal consistency across long recording sessions sits at the heart of what the music technology industry is bringing to AI teams right now.
For a broader look at how AI tools are reshaping the musician’s creative process, our AI for Musicians guide covers the spectrum, from production-workflow tools to open-source voice conversion models.
We trained a voice conversion model for this purpose. The goal: convert a specific speaker’s clean vocals into the same speaker’s distressed, hoarse delivery. Seed-VC is a strong fit for this kind of AI voice conversion: it performs well in zero-shot setups and provides a reliable starting point for fine-tuning.
For the task, we utilize a custom dataset of one speaker. It includes several takes per song to represent cleaner and more distressed vocals. The audio tracks were split into short (up to 20s) chunks that are processed so the silence duration is minimal.
Mel spectrograms of the same segment. Gold sample is ideally clean, Distressed sample represents hoarse vocals.
The Seed-VC paper introduces two baseline flow-matching models for our task. The smaller one is designed and trained for voice conversion tasks. It extracts the source sample semantic features and target speaker style embeddings. Optionally, it uses the target mel spectrogram and semantic features for refined conversion.
The larger Seed-VC model is trained for singing voice conversion. Additionally, it has conditioning on the source fundamental frequency features to match singing vocals better. The source semantic features are aligned with the acoustic features via a length regularization module of the model.
For the task of making the singing vocals more distressed, the singing voice conversion model is a better match. We evaluated both models in a zero-shot setup on our data. The SVC model keeps the fundamental frequency of the source vocals and the conversion results sound melodically and clean. Thus, we focused on fine-tuning this model to make its result more distressed.
How Does the Seed-VC Training Pipeline Work?
Extracted from the source utterance content features often include residual timbre information about the source speaker. To avoid the speaker timbre leakage, the Seed-VC training pipeline has an interesting modification. They use an external model as a timbre shifting model. So, during the training step, the input source waveforms are converted to simulate another speaker. The model input during training consists of the following parts:
- target speaker style embeddings;
- target Mel spectrogram with masked condition prompt region;
- concatenated original semantic and altered semantic features aligned with acoustic features.
The loss is calculated on the masked (source) region only.
In practice, the Seed-VC training pipeline simulates a scenario where the speaker’s timbre shifts abruptly mid-segment, sometimes mid-word. The model learns to generate the full segment using the timbre captured in its opening portion.
But why do we need simulation if we can have perfectly aligned actual speaker vocals? We tried to replace the timbre shifted part of the training DiT input with the actual clean vocals and the original waveforms with the actual distressed ones.
Fine-tuning the model in these setups showed unstable loss and decrease in metrics. It made the vocals sound more hoarse but they were slightly intermittent and still not distressed enough, so we moved to the next pipeline modification.
Become More Specific About Speaker
Since we are training a speaker-specific model to change the vocal distortion, we can make the training pipeline more speaker-oriented. We can remove the prompt audio conditioning and freeze the speaker embeddings during both training and inference. We can also fine-tune the model on the distressed (reference) audio samples only.
Unfortunately, removing conditioning on the target Mel spectrogram and semantic features does not work well. It makes the conversion result louder and more noisy but does not remove the desired vocal distortion. So it still can be applied.
For freezing the speaker style embeddings, the good approach was to take the mean embeddings of the distressed vocals of the singer.
The reference-only training worked very well. It resulted in a stable decreasing trend in the loss history since now we compute loss on the full input spectrogram rather than on its small part. The vocals after fine-tuning in the reference-only setup sound much more distressed compared to the previously tried fine-tuning approaches.
Key result: Reference-only training on distressed samples produced the most stable loss curve and the most convincingly distressed output, outperforming every setup that mixed in clean vocals.
For further experimentation, we tried a couple of techniques from the YingMusic-SVC paper. This paper describes training the Seed-VC singing voice conversion model with interesting approaches.
YingMusic-SVC Fine-Tuning Approaches
Keep the Balance: Frequency-Aware Loss Balancing
To address the imbalance in energy across frequency bands, the YingMusic-SVC paper introduces weighting coefficients for the loss function. The coefficient values increase with the frequency height and diffusion timestep. They also depend on the variance of the ground-truth diffusion velocity field.
Figure from the YingMusic-SVC paper representing the energy-balanced flow matching loss with time- and frequency-dependent weighting
Applying the loss balancing technique to our training pipeline significantly improved the fundamental frequency correlation between both the source/reference and converted audios. We tried using it with the Seed-VC native L1 loss, but it worked better when applying the L2 loss function.
Training tip: Frequency-aware loss balancing works better with L2 loss than L1. Start introducing weight coefficients from slightly lower frequencies than you might expect. The high-frequency ratio has an outsized effect on results.
The difference is audible. Below is a before/after comparison: the original clean vocal alongside the model output after reference-only training with frequency-aware loss balancing:
| Before: clean vocal (Gold) | After: distressed model output |
|---|---|
Timbre Adaptation Technique
Figure from the YingMusic-SVC paper representing the F0-aware timbre adaptor that refines global timbre embeddings into fine-grained, pitch-sensitive representations
In our experiments, using the timbre adaptor led to lower background noise in audio but decreased the fundamental frequency correlation. Combining the loss balancing strategy with the timbre adaptor helps improve the F0 metrics. So, these techniques compromise each other. But results of the model trained with loss balancing only are still better.
Style residuals cause converted samples to sound choppy: “ni-i-i-ight” instead of “niiiight”. With limited training data, the model runs out of steps to adapt the residuals before overfitting kicks in. More data is the direct fix.
Reinforcement Learning: Move Directly to the Goal?
The standout addition from YingMusic-SVC is a refining stage built on online reinforcement learning, specifically Flow-GRPO, extended with Stochastic Timestep Selection to keep noise influence more predictable. This stage runs after the regular fine-tuning phase.
The reinforcement learning with Flow-GRPO is done in two stages. During the sampling stage, the diffusion trajectories are sampled a couple of times per prompt with noise introduction at some single step. These trajectories are evaluated with non-differentiable rewards on the final step of the trajectory. Probabilities of the diffusion timesteps are computed during the training stage. The loss function includes the perceptual reward and KL divergence loss components.
Our experiments show that Flow-GRPO raises waveform amplitude and introduces more background noise, with a drop in metric values overall. With a small dataset, a single reward outperforms a combination of several: the model responds better to one clear direction than to competing signals.
Practical note: Flow-GRPO needs a larger dataset and well-defined target metrics to deliver. With limited data, one focused reward beats a combination of several. Keep the feedback simple.
Introducing a higher coefficient for the KL divergence loss influence increases the F0 correlation between the reference and converted audio samples but this does not affect the converted audio perceptual quality.
As for any generative AI task, the main headache for our goal is the metrics. We evaluated models with DNSMOS, UTMOS and SECS scores. But these metrics do not reflect the target distortion in vocals. The target distortion results in lower DNSMOS BAK and UTMOS values. But the same does the undesired noise. Applying SECS as the reward depends on how the speaker embedding extraction model reflects the difference between the clean and distressed vocals for the same speaker.
Flow-GRPO is effective but needs a larger dataset and well-defined target metrics to deliver on its potential.
Conclusion
The singing voice conversion task is not as simple as it may seem. It requires intricate techniques, enough data amount, and irregular metric evaluation.
Our experiments show the best model performance in setup applying the loss balancing strategy for training on the reference (distressed) samples only. We simplified the model making it more speaker-specific so it does not require the reference audio samples of distressed vocals for the target speaker.
Flow-GRPO remains one of the more promising directions, but it needs more data and compute to deliver on that promise. There is always room for further experimentation. Finding the right metrics is as much part of the work as the modelling itself.
If your project involves voice conversion, singing style transfer, or any other audio AI challenge (a music product, voice cloning pipeline, or live performance tool), IT-JIM’s audio AI development team can help assess the architecture, data requirements, and path to production.
Wondering what maximum vocal distortion sounds like without a model in the loop? Here is one hour of Jimmy Barnes doing it the hard way. Consider it the upper bound.
