Restoring speaker voices with zero-shot cross-lingual voice transfer for TTS

Vocal characteristics contribute significantly to the construction and perception of individual identity. The loss of one's voice, caused by physical or neurological conditions, can result in a profound sense of loss, striking at the very heart of one's identity. Speakers with degenerative neural diseases, such as amyotrophic lateral sclerosis (ALS), Parkinson's, and multiple sclerosis, may experience a degradation of some of the unique characteristics of their voice over time. Some individuals are born with conditions, like muscular dystrophy, that affect the articulatory system and limit their ability to produce certain sounds. Profound deafness also impacts vocal and articulatory patterns due to the absence of auditory input and feedback. These conditions present lifelong challenges in matching the typical speech heard widely.

In recent years, there have been new advances in voice transfer (VT) technology, integrated in text-to-speech (TTS), voice conversion (VC), and speech-to-speech translation models. For example, in our previous work, we built a VC model that converts atypical speech directly to a synthesized predetermined typical voice that can be more easily understood by others. Yet for many individuals with dysarthria, VT extends speech technologies to help them regain their original voice and potentially predict speech patterns they have lost.

A VT module can be designed for a given speaker using either few- or zero-shot training. In few-shot training for VT, a sample of speech from a given speaker is used to adapt a pre-trained model to transfer or clone their voice. This approach typically produces high quality speech with high speaker-voice fidelity, depending on the amount and quality of the training samples. A more challenging approach is zero-shot, which does not require training, but rather feeds audio reference samples (e.g., 10 seconds) from a given speaker to the system during generation, to transfer their voice into the output synthesized speech. These systems vary significantly in their quality and do not guarantee to produce high fidelity voices to the reference voice. Few-shot approaches can be effective for those speakers who once had typical speech and have banked a set of high quality samples of their voice before an etiology has progressed (or a physical injury has occurred). On the other hand, zero-shot is more appropriate for those dysarthric speakers who have not banked sufficient samples of their voice or have never had a typical voice. Moreover, a zero-shot system can be easily scaled and deployed.

In this blogpost, we describe a zero-shot VT module that can be easily plugged into a state-of-the-art TTS system to restore the voices of input speakers. It can be used both when speakers have banked a small set of their voice or when atypical speech is the only data available. We add this module to our TTS system and use it to restore the voices of speakers who banked their typical speech. We also show that the same model produces high quality speech with high fidelity voice preservation even when the input reference is atypical, useful for those who have not banked their voice or never had typical speech. Finally, we demonstrate that such a module is capable of transferring voice across languages, even though the language of the input reference speech is different from the intended target language.

We have done a subjective similarity evaluation of 500 pairs of audio, where each pair includes reference audio and synthesized speech of sentences generated by an LLM (using Gemini API). Each pair is presented to five human raters who are asked whether the two samples are spoken by the same person. We found that on average 76% (± 3.6) of those raters indicate that the two samples came from the same speaker.

The transcript was obtained using Dimitri’s personalized ASR model described in our previous work. Since Dimitri can’t hear this video, we asked 10 subjects who have a working relationship with Dimitri to score (1 to 10) of how similar the output voice is to that of Dimitri’s. We obtained an average of 8.1/10 (± 1.1).

As a second case study, we worked with Aubrie Lee, a Googler and advocate for disability inclusion who has muscular dystrophy, a condition that causes progressive muscle weakness and sometimes impacts speech production. Like Dimitri, Aubrie has never had a typical speech. Similar to our procedure above, using our zero-shot model, we give the model about 14 seconds of atypical reference speech and use the model to synthesize the automatic transcript of the following video.

We asked Aubrie to score how similar she thinks the output voice is to hers and she gave it 8/10.

We recognize that in the context of voice transfer technology, the potential for misuse of synthesized speech is a growing concern. To address this, we use audio watermarking to embed watermarks so synthesized speech from our model can be detected. This technique involves embedding imperceptible information within the synthesized audio waveform. This hidden data can be detected using specialized software, enabling the identification of potentially manipulated or misused audio content. It's important to note that the risk of misuse is significantly lower for individuals who have never had typical speech patterns. In such cases, the synthesized nature of the output would be readily apparent, minimizing the potential for deception.

We would like to express our gratitude to Aubrie Lee, Dimitri Kanevsky, Kyle Kastner, Isaac Elias, Gary Wang, Andrew Rosenberg, Takaaki Saeki, Yuma Koizumi, Bhuvana Ramabhadran, Françoise Beaufays, RJ Skerry-Ryan, Ron Weiss, Zoe Ortiz, and the rest of Google’s speech team for their helpful feedback and contributions to this project.