Cross-Lingual Knowledge Distillation via Flow-Based Voice Conversion for Robust Polyglot Text-to-Speech

Dariusz Piotrowski; Renard Korzeniowski; Alessio Falai; Sebastian Cygert; Kamil Pokora; Georgi Tinchev; Ziyao Zhang; Kayoko Yanagisawa

doi:10.1007/978-981-99-8126-7_20

Cross-Lingual Knowledge Distillation via Flow-Based Voice Conversion for Robust Polyglot Text-to-Speech

Abstract

In this work, we introduce a framework for cross-lingual speech synthesis, which involves an upstream Voice Conversion (VC) model and a downstream Text-To-Speech (TTS) model. The proposed framework consists of 4 stages. In the first two stages, we use a VC model to convert utterances in the target locale to the voice of the target speaker. In the third stage, the converted data is combined with the linguistic features and durations from recordings in the target language, which are then used to train a single-speaker acoustic model. Finally, the last stage entails the training of a locale-independent vocoder. Our evaluations show that the proposed paradigm outperforms state-of-the-art approaches which are based on training a large multilingual TTS model. In addition, our experiments demonstrate the robustness of our approach with different model architectures, languages, speakers and amounts of data. Moreover, our solution is especially beneficial in low-resource settings.

Citations

0

CrossRef
0

Web of Science
0

Scopus

Authors (8)

Dariusz Piotrowski
Renard Korzeniowski
Alessio Falai
Sebastian Cygert dr inż.
Kamil Pokora
Georgi Tinchev
Ziyao Zhang
Kayoko Yanagisawa

Cite as

Full text

full text is not available in portal

full content of the article see on external site open in new tab

Keywords

NEURAL TEXT-TO-SPEECH MULTILINGUAL SYNTHESIS VOICE CONVERSION SYNTHETIC DATA NORMALISING FLOWS

Details

Category:

Conference activity

Type:

materiały konferencyjne indeksowane w Web of Science

Language:

English

Publication year:

2023

Bibliographic description:

Piotrowski D., Korzeniowski R., Falai A., Cygert S., Pokora K., Tinchev G., Zhang Z., Yanagisawa K..: Cross-Lingual Knowledge Distillation via Flow-Based Voice Conversion for Robust Polyglot Text-to-Speech, W: , 2023, ,.

DOI: