Editing State-of-the-art (section)

== Language-specific Text-To-Speech ==

=== Introduction ===
State-of-the-art Text-to-Speech systems have different performances based on the language they are developed for and trained on. We choose to focus on language-specific TTS and provide a review of state-of-the-art techniques to synthesise languages other than English. Even though this does not necessarily restrict to Low-Resourced Languages (LRLs), we decided to focus mainly on techniques developed for LRLs, and more broadly, approaches that entail the use of a limited amount of data.

The article summaries below include the topics of multilingual data strategies, TTS with phonological features, and Transfer Learning.

=== Article summaries ===

==== Do, P., Coler, M., Dijkstra, J., & Klabbers, E. (2021). A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages. Proc. Interspeech 2021, 16–20. doi: 10.21437/Interspeech.2021-1565 ====

* Summary: The article provides an overview of strategies for test-to-speech for low-resource langauges, focusing on Multilingual Data strategies. More specifically, this article presents an evaluation of the results of the previous studies on LRLs TTS, an evaluation of the influence of data augmentation techniques employed on the performance of the models and the proposal of a new measure to evaluate the performance of multilingual vs. monolingual systems with different evaluation metrics, namely MultiLingual Model Effect (''MLME''). The performance of the strategies analysed is also checked by verifying how different factors influence it.
* RQ: 
*# Using the same limited amount of LRL data, how does the output quality of multilingual TTS models compare to that of monolingual models?
*# What factors in the data augmentation strategy influence the effect of using multilingual TTS models on output quality, and to what extent do they affect it?
* Hypothesis: Looking at the correlations between data augmentation strategies and synthesized speech quality, tools that use multilingual data can be provided for future research in TTS for LRLs, especially regarding the efficiency of using such data.
* Conclusion: Multilingual approaches are more effective in training for LRLs. The factors that affect the performance are: 
** target language data ratio between corresponding multilingual and monolingual models;
** target language data balance ratio over total training data
** amount of target language data.
* Critical observations: The paper only focuses on multilingual data strategies, and justifies the choice by saying that multispeaker data are harder to collect for LRLs. Even though I understand the reasoning behind this, I believe this is not entirely true. On one hand, it is true indeed that it is harder to find many speakers for a LRLs, since oftentimes such languages are also minority languages. On the other hand, collecting multispeaker data means that each speaker can contribute with a very small amount of data and still get enough of them. This means that by adopting multispeaker TTS techniques, we don't need to record one speaker for a long time, but rather multiple speakers for a short time. This multi-speaker approach, I believe, could be used in combination with Transfer Learning to improve the results of LRLs TTS systems, even though this implies adding complezity to the pipeline.
* Relevance: The most relevant outcome of this study, especially for LRLs TTS, is that the '''''language family is not relevant for the selection of the target-source language pair''', no matter the architecture.'' In my opinion, the conclusions of this paper are also relevant for medium-resourced languages and in general for the synthesis of non-standard speech and for all the types of speech that are not widely covered by the research so far. 

==== Staib, M., Teh, T. H., Torresquintero, A., Mohan, D. S. R., Foglianti, L., Lenain, R., & Gao, J. (2020). Phonological features for 0-shot multilingual speech synthesis. ''arXiv preprint arXiv:2008.04107''. ====

* Summary: This article primarily aims to utilize a limited set of phonological features (PF), derived from the International Phonetic Alphabet (IPA), for achieving 0-shot speech synthesis and code-switching within a monolingual model. Specifically, the study selects Tacotron 2 as the baseline for comparison against methods of random initialization (RANDOM), manual mapping (MANUAL), and the PF-based approach (AUTO) proposed in this work. The conclusion drawn is that the speech generated using the AUTO method is more comprehensible.
* RQ: The research question of this paper explores whether phonological features (PF) can facilitate speech synthesis for languages not seen during training. Additionally, it examines whether PF can facilitate code-switched speech synthesis.
* Hypothesis: The hypothesis of the article is that phonological features (PFs), derived from the International Phonetic Alphabet (IPA), can enable 0-shot text-to-speech (TTS) synthesis and code-switching in languages that are not seen during training, even within monolingual models.
* Conclusion: The conclusion of the article is that by replacing the character input in Tacotron 2 with phonological features, a model topology can be created that is language-independent and allows for the automatic approximation of sounds unseen in training. The study shows that phonological features (PFs) can not only facilitate zero-shot speech synthesis in untrained languages within a small multilingual or even a monolingual model but also facilitate the synthesis of sounds that completely unseen in training. This suggests potential applications in code-switching and TTS for low-resource languages.
* Critical observations: This article mainly addresses the problem of 0-shot speech synthesis and code switching by using phonological features (PF). The significant advantage of this method is that it can reduce the amount of data for training multi-language speech synthesis models, and it is very helpful in low-resource languages and code-switched TTS. But PF may not capture all the differences of a language, especially for those with unique phonetic and phonological features, and the selected PFs might not adequately represent these languages. In addition, this article focuses more on generating understandable speech and may ignore the importance of features such as prosody.
* Relevance: This paper is mainly related to the fileds of cross-language speech synthesis and code switching speech synthesis. Some other studies have also proposed to find a unified representation (such as Unicode) to replace phoneme or text to achieve cross-language synthesis, but the PF proposed in this paper may be A better choice because these features retain speech features to a certain extent and help the model learn better.

==== Do, P., Coler, M., Dijkstra, J., & Klabbers, E. (2023). Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection. ''arXiv preprint arXiv:2306.12040''. ====

*Summary: This paper compares two methods in TTS for low-resource languages: PHOIBLE-based phone mapping and phonological features input. Various languages are tested to see how these methods work across different languages. The findings show that both methods improve speech quality, with phonological features performing better. The study also examines two criteria for choosing source languages: Angular Similarity of Phone Frequencies (ASPF) and language family tree distance. ASPF is found effective, especially with phone-based input, while the language distance criterion does not yield expected results.
* RQ: The paper aims to explore how to most effectively deal with the input mismatch between languages and how to select the best source language to improve output quality in TTS for low-resource languages.
* Hypothesis: 
*# Transfer learning using PHOIBLE-based phone mapping and phonological feature inputs can improve TTS output quality for low-resource languages.
*# Angular Similarity of Phone Frequencies (ASPF) is an effective criterion for selecting source languages, more so than traditional broad language family classification.
* Conclusion: 
*# Both phone mapping and feature inputs can enhance output quality, with feature inputs showing better performance, although the effectiveness depends on the specific language pairing.
*# ASPF is effective in selecting source languages, especially when using label-based phone inputs, while the distance based on the language family tree does not work as expected.
* Critical observations:
*# Although ASPF is effective in some cases, its effectiveness is not universal across all language combinations, indicating the need for further research to understand influencing factors.
*# The unexpected results with the language family tree distance suggest that there might be unidentified factors at play, necessitating further investigation.
* Relevance: This research is significant for the development of TTS technology for low-resource languages, especially in offering new insights into source language selection and handling input mismatches between languages. Moreover, the proposed methods are important for the multilingual applicability and scalability of speech technologies.

==== Wells D, Richmond K. Cross-lingual transfer of phonological features for low-resource speech synthesis[C]//Proceedings of the 11th Speech Synthesis Workshop, Budapest, Hungary. 2021: 160-165. ====

* Summary: In this paper, researchers compare two methods: fine-tuning phonemic representations and using phonological features. They used SPE-style phonological features, offering a binary representation of phonemes, which helps describe and analyze speech patterns in English and German. The study discovers that even with limited target language data, fine-tuning can generate speech comparable to models trained from scratch. Using phonological features slightly improves naturalness ratings compared to using phonemes alone. These findings highlight the practical benefits of phonological features in improving TTS output quality across languages.
* RQ: Does the use of different input representations (phonemes and phonological features) affect the naturalness of synthesized speech in text-to-speech synthesis using cross-lingual transfer learning?
* Hypothesis: In cross-lingual transfer learning for text-to-speech synthesis, the use of different input representations (phonemes and phonological features) affects the naturalness of synthesized speech.
* Conclusion: The study confirmed the effectiveness of cross-lingual fine-tuning for training synthetic voices with limited target language data. Phonological features were found to offer practical benefits over phonemes in terms of parameter sharing during transfer learning.
* Critical observations: There was a slight improvement in naturalness ratings when using PFs over phonemes. Future research may explore multilingual grapheme-to-phoneme systems and utilize additional linguistic resources to enhance low-resource pipelines for text-to-speech synthesis
* Relevance: Phonological features were found to offer practical benefits over phonemes in terms of parameter sharing during transfer learning, which can be applied greatly in LRLs TTS.

==== Synthesis ====
To summarize, text-to-speech research in recent years has explored multilingual data strategies, phonological features, and transfer learning methods to enhance its performance, especially for low-resource languages. 

Based on the studies reported above, multilingual models outperform monolingual ones, showing promise in improving the voice quality with limited data. Moreover, phonological features facilitate zero-shot synthesis and code-switching, benefiting LRLs and cross-language applications. Transfer learning methods like PHOIBLE-based phone mapping and phonological feature inputs improve output quality, with ASPF effective for source language selection. Finally, fine-tuning phonological representations enhances speech naturalness, suggesting the potential for multilingual g2p systems.

These findings emphasise innovative approaches' importance in advancing TTS, with a specific focus on LRLs, offering insights into effective strategies and criteria for synthesis quality and scalability.

=== Contributors ===
* Article Do, et al. (2021) 'A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages': Alice Vanni
* Article Staib et al. (2020) 'Phonological features for 0-shot multilingual speech synthesis': Wang Yinqiu
* Article Do, et al. (2023) 'Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection': Annie Zhou
* Article Wells D, et al. (2021) 'Richmond K. Cross-lingual transfer of phonological features for low-resource speech synthesis': Ding
* Introduction: All
* Synthesis: All