Editing State-of-the-art (section)

== Theme: TTS naturalness ==

=== Introduction ===
TTS systems have significantly advanced over time, achieving remarkable intelligibility and near-human naturalness in synthetic voices through deep learning advancements. However, the naturalness of synthetic voice remains limited to sentences, and lacks the expressivity found in human conversation such as appropriate emotion, prosody and style. Despite these limitations, natural TTS, particularly expressive speech synthesis, plays a crucial role in achieving human-like speech and enhancing the engagement of synthesized speech. Moreover, it facilitates the broader adoption of TTS technology across various domains within the field of speech technology. In this context, our group focuses on the theme of TTS naturalness with two interconnected subthemes: exploring advanced models and relevant theories. By addressing these subthemes, we aim to provide a comprehensive overview of the current state-of-the-art in TTS naturalness.

=== Article summaries ===

==== Subtheme 1: State-of-the-art Models ====

===== Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., Soong, F., Qin, T., Zhao, S., & Liu, T.-Y. (2022). NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality. ''arXiv preprint arXiv:2205.04421''. =====
* Summary: NaturalSpeech proposes a system for converting text to speech (TTS) that achieves human-level quality. It leverages a variational autoencoder (VAE) to bridge the gap between text and speech waveforms.
* RQ (Research Question): Can a TTS system achieve speech quality indistinguishable from humans?
* Hypothesis: By incorporating a VAE and specific techniques to improve the model's understanding of text and speech features, NaturalSpeech can generate speech indistinguishable from humans.
* Conclusion: The paper argues that NaturalSpeech achieves human-level speech quality based on statistical measures (MOS and CMOS) in human evaluations.
* Critical Observations: The evaluation relies on subjective human ratings, which might be influenced by factors beyond speech quality.The research focuses on a single benchmark dataset, limiting generalizability.The paper doesn't explore how NaturalSpeech performs on diverse speaking styles or accents.

* Relevance: This is related to my study because it provides a definition of human-level quality, and this particular model has achieved the highest Mean Opinion Score (MOS) recorded thus far. Hence, I am considering using this model as a basis for my study.

===== Kong, J., Kim, J., & Bae, J. (2020). Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. ''Advances in neural information processing systems'', ''33'', 17022-17033. =====

* Summary: This article introduces HiFi-GAN, a model that can efficiently synthesize high-quality speech audio. HiFi-GAN consists of a generator and two discriminators: multi-scale discriminator and multi-period discriminator. Improve training stability and model performance by adversarially training the generator and discriminator and using two additional loss functions.

* RQ:Can HiFi-GAN effectively synthesize high-quality speech audio with computational efficiency comparable to human-level synthesis, while also demonstrating generalization across speakers and adaptability to various configurations?
* Hypothesis:By leveraging the characteristic patterns of speech audio and designing a discriminator to capture these patterns effectively, it is possible to develop a speech synthesis model, HiFi-GAN, that outperforms existing models in terms of synthesis quality and speed.
* Conclusion:HiFi-GAN significantly advances speech synthesis by efficiently generating high-quality audio, surpassing existing models in both synthesis quality and speed. By leveraging speech audio patterns and a carefully designed discriminator, this model demonstrates robustness across various scenarios, including unseen speakers and noisy inputs, while offering potential for on-device natural speech synthesis with low latency and memory requirements. Additionally, the flexibility of generator configurations enhances adaptability without the need for extensive hyper-parameter search.
* Critical observations:Due to the wide application of HiFi-GAN technology in the field of speech synthesis, there may be some ethical or social impacts, including concerns related to voice cloning, privacy and false information.
* Relevance:This paper is closely related to the topic of non-language-specific text-to-speech, as it demonstrates a breakthrough in HiFi-GAN models in synthesizing high-quality speech, with generalization capabilities, and the ability to handle inputs of different languages and speaking styles.

===== Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., ... & Zhao, Z. (2023, July). Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In ''International Conference on Machine Learning'' (pp. 13916-13932). PMLR. =====
* Summary: The article introduces "Make-An-Audio," a system utilizing a prompt-enhanced diffusion model for TTS generation, aiming to improve the naturalness and expressiveness of synthesized audio.
* RQ: How does the model improve the naturalness of TTS?
* Hypothesis: By introducing pseudo prompt enhancement and spectrogram autoencoders, the model can effectively utilize unsupervised language-free data and higher-level semantic understanding to enhance the naturalness and expressiveness of speech synthesis.
* Conclusion: "Make-An-Audio" successfully enhances the naturalness and expressiveness of speech synthesis, achieving state-of-the-art performance in evaluations.
* Critical observations: The performance of "Make-An-Audio" is still partly dependent on extensive data and complex model training. In addition, there is still space for improvement in expressing the emotions and rhythms of human conversations.
* Relevance: The "Make-An-Audio" system presented in the paper offers an effective solution to the limitations in naturalness and expressiveness currently faced by TTS

==== Subtheme 2: State-of-the-art Theories ====

===== Noufi, C., May, L., & Berger, J. (2023). Context, Perception, Production: A Model of Vocal Persona. ''PsyArXiv. July'', ''28''. =====
* Summary: This article introduces a contextualized production-perception model of vocal persona, developed through qualitative analysis of interviews with voice and performance experts. It emphasizes the influence of context on an individual's vocal expression, reflecting the intricacies of human communication.
* RQ: What is the relationship between context, vocal expression, and identity?
* Hypothesis: It is a qualititative study and does not hve a formulated hypothesis.  Instead of attempting to falsify a hypothesis as in most quantitative studies, it explores answers to the research question through thematic analysis.
* Conclusion: Speakers actively select different vocal personas and adjust relevant vocal expressions in response to the surrounding context, facilitating a transition in the perception of persona.
* Critical observations: The proposal of the vocal persona model and general conclusions are based on interviews with 21 voice and performance experts, which may have limitations in terms of subjective bias and generalizability beyond this specific context.
* Relevance: This study underscores the necessity for speakers to adapt their speaking styles to accommodate different social contexts, highlighting the significance of context in vocal expression. It proposes the incorporation of vocal persona into expressive vocal synthesis with a three-spoke model and a framework for persona-guided vocalization, enriching the framework of TTS naturalness and expressiveness.

===== Vainer, J., & Dušek, O. (2020). Speedyspeech: Efficient neural speech synthesis. ''arXiv preprint arXiv:2008.03802''. =====
* Summary: This paper introduces a novel student-teacher network architecture called "SpeedySpeech" for fast and high-quality neural speech synthesis. The system is designed to enable faster-than-real-time speech synthesis while requiring minimal computing resources, and deliver audio quality that is superior to existing models such as the Tacotron 2. The model uses the teacher network for duration extraction, the student network for spectrogram synthesis, and combines it with the MelGAN vocoder to output high-quality audio. The training process is efficient and can be completed in less than 40 hours on a single 8GB GPU. 
* RQ: How can we develop a neural speech synthesis system that does not require extensive computing resources while maintaining fast training times, fast inference, and high-quality audio output?
* Hypothesis: Assuming a student-teacher network architecture with simplified convolutional blocks and only a single attention layer in the teacher model, it is possible to surpass existing models in terms of training efficiency and audio quality while maintaining fast inference.
* Conclusion: The proposed SpeedySpeech model successfully achieves its goals by demonstrating that self-attention layers are not necessary for high-quality audio generation and that simpler, fully convolutional methods enable a more efficient training process and faster synthesis. The model's speech quality score is significantly higher than Tacotron 2, and it can be trained efficiently on a single GPU and even run in real time on the CPU.
* Critical observations: The article proposes ways to address the trade-off between training efficiency and audio quality in neural speech synthesis. By using only a single attention layer in the teacher model and eliminating sequence generation in the student network, the authors achieve important simplifications that increase model efficiency. In the model evaluation, the authors comprehensively considered objective indicators (such as MAE, SSIM) and subjective listening tests to provide a comprehensive assessment of model performance.
* Relevance: This speech synthesis model has applications in many fields, including virtual assistants, machine translation, etc. The SpeedySpeech model can synthesize speech in real time on moderate hardware, making it particularly suitable for deployment in resource-constrained environments. Additionally, the focus on efficiency and quality sets new benchmarks for future research and development in this area.

===== Peiró-Lilja, A., & Farrús, M. (2020). Naturalness Enhancement with Linguistic Information in End-to-End TTS Using Unsupervised Parallel Encoding. ''Interspeech 2020'', 3994–3998. <nowiki>https://doi.org/10.21437/Interspeech.2020-1788</nowiki> =====
* Summary: The paper explores enhancing the naturalness of synthesized speech in E2E-TTS systems by incorporating linguistic features like POS tags and punctuation into the Tacotron 2 model, aiming to improve prosody to resemble human speech more closely.
* RQ: How can linguistic information be integrated into the Tacotron 2 system to improve the naturalness of synthesized speech prosody?
* Hypothesis: The hypothesis is that by embedding POS tags and punctuation locations as additional linguistic features into the Tacotron 2 system, the synthesized speech will exhibit improved naturalness and prosody, making it more similar to human speech.
* Conclusion: The study concludes that the incorporation of linguistic features through a parallel encoder significantly improves the naturalness of synthesized speech. The authors proposed two different architectures for the parallel encoder: one based on convolutional and recurrent layers (2DConv+BiLSTM) and another composed of bidirectional recurrent and linear layers (BiGRU+Linear). Both architectures aimed to process the binary matrix representing POS tags and punctuation locations. The results from objective tests and perceptual evaluations indicated that the model with the 2DConv+BiLSTM parallel encoder performed the best in terms of naturalness, as it more closely matched human pitch contours and overall speech quality.
* Critical observations: Critically, the paper notes that while both parallel encoder architectures showed improvements over the Tacotron 2 baseline, the 2DConv+BiLSTM version provided better results in terms of naturalness. However, it also introduced a slight increase in Mel Cepstral Distortion (MCD), suggesting a trade-off between naturalness and certain acoustic quality metrics. The BiGRU+Linear model, despite being lighter and faster, underperformed in perceptual tests, possibly due to its reduced complexity and higher cepstral distortion.
* Relevance: The findings of this research are relevant for the development of more natural and human-like E2E-TTS systems, which have applications in various domains such as automatic dialogue systems, storytelling, and voice assistants. By enhancing the prosody of synthesized speech, these systems can provide more engaging and realistic interactions, improving user experience and accessibility. Furthermore, the study contributes to the broader field of speech synthesis by demonstrating the potential of unsupervised parallel encoding of linguistic features to improve speech naturalness.

===== Cai, X., Dai, D., Wu, Z., Li, X., Li, J., & Meng, H. (2021). Emotion Controllable Speech Synthesis Using Emotion-Unlabeled Dataset with the Assistance of Cross-Domain Speech Emotion Recognition. ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5734–5738. <nowiki>https://doi.org/10.1109/ICASSP39728.2021.9413907</nowiki> =====
* Summary: This article proposes an approach for emotional TTS synthesis on a dataset without emotion labels, using a cross-domain speech emotion recognition model and an emotional TTS model, hoping to express similar results in specific emotional expressiveness and speech quality compared to models with emotion labels.
* RQ: Can we use the achievements and features of SER to solce the problem of lack of emotion-annotated dataset for emotional TTS?
* Hypothesis: By performing mean opinion score evaluations and emotion recognition perception evaluation in 4 emotion categories and 2 polarities of emotion dimensions, our GST-based model can generate speech with expected emotions, while trained on a fully emotion-unlabeled dataset. 
* Conclusion: Through comparing their created cross-domain model with a baseline model, they found that both their 4-categorical model and 2-dimensional model almost achieve as good speech quality as the baseline system, with higher p-values than the significance level of 0.05, indicating no significant differences. Furthermore, they found that both categorical models, one trained on the utterances with highest posterior (top-K scheme) and one trained on the full set of audio, revealed an overall higher accuracy than the baseline model, at 78.75% and 49.25%, respectfully, compared to the baseline 36.75%, which indicated their cross-domain model and top-K scheme were effective in emotional expressiveness. 
* Critical observations: The choice to use a top-K scheme, as described earlier, is interesting to offset the number of mispredictions made by the SER model, as the SER model is far less reliable that humans. By choosing to use the more reliable audio set, there could be an argument that their choice could inflate their results. However, taking this into account, they did in fact train the model on the full, unaltered, set of audio and still returned a higher accuracy than the baseline model. The fact that this shows promise in returning accurte and sufficient quality in emotional speech synthesis with unlabeled emotion datasets brings promise to a possible increase in speed and efficiency for training other models. 
* Relevance: The proposed approach, in the authors' words, greatly reduces the threshold of emotional synthesis in regard to amotion-annotated data, reducing the time, cost, and relevant quality of the speech data needed for emotional TTS systems. 

=== Synthesis ===
From the articles on non-language-specific text-to-speech (TTS) synthesis highlights several emerging trends and debates within the field of voice technology. The articles reviewed contribute to a comprehensive understanding of the state-of-the-art in TTS naturalness, spanning advanced models and theories that aim to bridge the gap between synthetic and human speech.

'''Emerging Trends:'''

1. Advancements in Model Architecture: A significant trend is the development of advanced TTS models, such as NaturalSpeech, Make-an-Audio, HiFi-GAN, and SpeedySpeech, which leverage innovative techniques like variational autoencoders, prompt-enhanced diffusion models, adversarial training, and efficient network architectures. These models aim to improve the naturalness and expressivity of synthetic speech, achieving closer approximation to human speech quality.

2. Integration of Linguistic and Emotional Information: There is a growing emphasis on incorporating linguistic features and emotional expressivity into TTS systems. Studies like the one by Peiró-Lilja & Farrús, and Cai et al. demonstrate the potential of enhancing speech naturalness and emotional expressivity by embedding linguistic cues and leveraging emotion-unlabeled datasets with cross-domain speech emotion recognition models. This approach points to a nuanced understanding of speech production, where prosody, context, and emotional tone play crucial roles.

3. Exploration of Vocal Persona and Contextual Factors: The study by Noufi, May, & Berger introduces the concept of vocal persona, highlighting the influence of context on vocal expression and identity. This reflects an acknowledgment of the complexity of human speech, where individuals adapt their vocal style to different social contexts. Integrating such contextual and persona-based nuances into TTS systems could lead to more sophisticated and contextually aware speech synthesis.

'''Debates:'''

Quality vs. Complexity: Despite advancements, a recurring challenge is the trade-off between improving speech quality and managing the complexity and computational demands of TTS models. Models like HiFi-GAN and SpeedySpeech address this by optimizing for efficiency and fidelity, yet questions remain about the balance between model simplicity and the ability to capture the rich variability of human speech.

In conclusion, the field of TTS is witnessing rapid advancements and facing complex challenges. The synthesis of findings from the reviewed articles underscores the importance of multidisciplinary approaches that integrate technical innovations with insights from linguistics and psychology to advance towards more natural, expressive, and ethically developed TTS technologies.
=== Contributors ===
Contributors: A list of contributors by contribution 

* Article Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models: Yilan Wei
* Article Context, Perception, Production: A Model of Vocal Persona: Chenyi Lin
* Article NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality: Yi Lei
* Article HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis: Yanhua, Liao
* Article Naturalness Enhancement with Linguistic Information in End-to-End TTS Using Unsupervised Parallel Encoding: Jingxuan Yue
* Article Emotion Controllable Speech Synthesis Using Emotion-Unlabeled Dataset with the Assistance of Cross-Domain Speech Emotion Recognition: Jocomin Galarneau
* Article SpeedySpeech: Efficient Neural Speech Synthesis: Weihao Jiang
* Introduction: Chenyi Lin
* Synthesis: Yi Lei