Real-time speech-to-speech translation

Real-time communication is an integral part of both our professional and personal lives. When speaking to people remotely across language barriers, it can be difficult to truly connect by just relying on state-of-the-art translated captions, as they lack personality and real-time responsiveness essential for fluid conversation. The arrival of speech-to-speech translation (S2ST) bridges this gap by directly generating translated audio, leading to more natural communication. Existing speech-to-speech translation systems often incur significant delays (4–5s), tend to accumulate errors, and typically lack personalization.

Today we describe an innovative end-to-end S2ST model that overcomes these limitations, enabling live translation in the original speaker's voice with only 2 second delay. The novel architecture leverages a streaming framework and, with training on time-synchronized data, significantly reduces the delay between the original input and the translated speech. To support a breadth of languages, we introduce a scalable time-synced data acquisition pipeline that allows us to gradually expand the system to include more languages. This technology has demonstrated its effectiveness through successful deployment in real-time sensitive use cases.

Cascaded S2ST

Prior real-time speech-to-speech technologies employed a cascaded pipeline of individual processing blocks:

1. Firstly, the source audio is transcribed to text using automatic speech recognition (ASR) AI models.
2. Next, the transcribed text is translated word-for-word to the target language using automatic speech translation (AST).
3. Finally, the translated text is converted back to audio using text-to-speech pipelines (TTS).

Despite the high quality of the individual cascade components, achieving a seamless, real-time S2ST experience has been challenging due to three primary factors:

1. Significant delays of 4–5 seconds, forcing turn-based conversations.
2. Accumulated errors at each stage of the translation process.
3. A notable lack of personalization due to the general-purpose TTS technology.

A novel end-to-end, personalized S2ST

To significantly advance S2ST, we created a scalable data acquisition pipeline and developed an end-to-end model that provides direct, real-time language translation with just a two-second delay:

1. Scalable data acquisition pipeline: We created a data processing pipeline to convert raw audio into a time-synchronized input/target dataset. This was achieved by integrating existing ASR and TTS technologies with precise alignment steps, ensuring translated audio best matches the input. Rigorous filtering and validation were employed to remove difficult-to-align examples.
2. Real-time speech-to-speech translation architecture: We introduced an audio-specific streaming machine learning architecture to support training on this time-synchronized data. Building on the AudioLM framework and fundamental transformer blocks, this architecture is designed to handle continuous audio streams, allowing the model to decide when to output translations. It is also structured to manage hierarchical audio representations using the SpectroStream codec technology.

Utilizing the three generated alignments from the previous steps, the overlap between them is calculated, yielding alignment masks between the source and target audio. These alignment masks are then used to guide the loss computation during training.

Invalid overlaps or translations that fail to meet delay requirements are filtered from the training dataset. The remaining aligned data is used to train the streaming S2ST model in chunks of up to 60 seconds. Various audio augmentation techniques are also applied during training, including sample rate reduction, reverberation, saturation, and denoising.

Real-time speech-to-speech translation architecture

The end-to-end S2ST model leverages fundamental transformer blocks and consists of two main components:

• Streaming encoder: Summarizes source audio data based on the preceding 10 seconds of input.
• Streaming decoder: Predicts translated audio autoregressively, using the compressed encoder state and predictions from previous iterations.

A feature of these models is their representation of audio as a 2D set of tokens, known as RVQ audio tokens. As shown below, the X-axis represents time, while the Y-axis represents a set of tokens that describe the current audio segment. When summed, all tokens in a specific set can be readily converted into an audio stream using an ML codec. The number of tokens controls the audio quality for every segment, with more tokens yielding higher fidelity. The model predicts tokens sequentially, prioritizing those at the beginning. Typically, 16 tokens are sufficient for high-quality audio representation of a 100 ms chunk.

The model outputs a single text token in addition to the audio tokens. This text token acts as an extra prior for audio generation and enables direct metric calculation (BLEU) without relying on proxy ASR systems.

During training, a per-token loss is applied to the model to ensure accurate translation. The model's prediction delay, or lookahead, can be adjusted by shifting ground truth tokens to the right, allowing for flexibility based on the target language's complexity. For real-time conversations, a standard 2-second delay is typically used, which is suitable for most languages. While a longer lookahead improves translation quality by providing more context, it negatively impacts the real-time communication experience.

In addition to the internal 2 second delay, the model's inference time contributes to the overall system latency. To minimize this and achieve real-time performance, we implemented several optimization techniques, including hybrid low-bit (int8 and int4) quantization and optimized CFG precomputation.

The examples of translation for different language pairs with developed models with corresponding ground truth (taken from publicly available CVSS dataset) follow:

The current end-to-end model delivers robust performance for five Latin-based language pairs (English to and from Spanish, German, French, Italian, Portuguese), enabling our initial product launches. We are also observing promising capabilities in other languages, such as Hindi, that we plan to develop further. Future enhancements will focus on improving the dynamism of the model's lookahead. This will enable the S2ST technology to seamlessly adjust to languages with word orders significantly different from English, facilitating more contextual rather than literal word-for-word translation.

We believe that this breakthrough in S2ST technology will revolutionize real-time, cross-language communication, turning a long-envisioned concept into reality.

Acknowledgements

We are sincerely grateful to everyone who contributed to this project; their critical contributions were instrumental in making it a reality. We are particularly thankful to our colleagues Kevin Kilgour, Pen Li, Félix de Chaumont Quitry, Michael Dooley, Jeremy Thorpe, Mihajlo Velimirović, Alex Tudor, Christian Frank, Daniel Johansson, Hanna Silén, Christian Schuldt, Henrik Lundin, Esbjörn Dominique, Marcus Wirebrand, Daniel Kallander, Pablo Barrera González, Huib Kleinhout, Niklas Blum, Fredric Lindstrom, Esha Uboweja, Karthik Raveendran, Frédéric Rechtenstein, Xing Li, Shengyi Lin, Cheng Yang, Jason Fan, Matsvei Zhdanovich, Jianing Wei, and Matthias Grundmann.