State-of-the-art

Theme: Template copy/paste but do not delete

Introduction

Briefly introduce your thematic focus and its significance in the field of speech technology.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Jones et al. 2023: YOUR NAME
• Article XXX: YOUR NAME
• Introduction: All
• Synthesis: All

Low-resource ASR

Introduction

Our theme focuses on automatic speech recognition (ASR) of low-resource languages. Low-resource languages are often underrepresented in ASR due to the limited amount of data, limited amount of speakers, and low commercial impact. However, it is important for both preserving and encouraging the use of low-resource languages to allow for users to utilize ASR for their own language. Therefore, our theme is significant in the field of speech technology. A similar data scarcity challenge exists for ASR of dysarthric speech, which occurs in neurodegenerative disorders like Parkinson's disease. Some transfer learning techniques has been explored to improve speech systems for dysarthric and low-resource languages respectively by leveraging data from other languages/domains. Such cross-domain transfer learning shows promise, but requires careful study to effectively bridge the data gaps.

Article summaries

Wang, S., Rohdin, J., Plchot, O., Burget, L., Yu, K., & Cernocky, J. (2020). Investigation of Specaugment for Deep Speaker Embedding Learning. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7139–7143. https://doi.org/10.1109/ICASSP40776.2020.905348

• Summary: The article investigates the effectiveness of SpecAugment, a data augmentation method, for speaker verification tasks using TDNN and ResNet34 models with Softmax and AAMSoftmax loss functions. Experiments on NIST SRE 2016 Cantonese and Tagalog subsets and Voxceleb1 dataset show improved performance with SpecAugment, achieving 3.72% and 11.49% EER for NIST SRE 2016 Cantonese and Tagalog, respectively, and 1.47% EER for Voxceleb1. SpecAugment demonstrates promising results for speaker verification across different languages, enhancing system robustness without complex offline augmentation.
• RQ: How effective is SpecAugment, a data augmentation method originally proposed for speech recognition, when applied to speaker verification tasks across different languages, specifically Cantonese and Tagalog?
• Hypothesis: Applying SpecAugment, a data augmentation technique initially developed for speech recognition, to speaker verification tasks will lead to performance improvements across different languages, including Cantonese and Tagalog.
• Conclusion: Implementing SpecAugment for speaker verification tasks yields significant performance improvements across different languages. Specifically, the study demonstrates that SpecAugment, applied on-the-fly without complex offline augmentation methods, achieves state-of-the-art results in speaker verification tasks for Cantonese and Tagalog, as well as for the Voxceleb1 dataset.
• Critical observations: The critical observation of the article focuses on the implementation of SpecAugment for speaker verification tasks across various languages, particularly Cantonese and Tagalog, which are considered low-resource languages. The study demonstrates that SpecAugment, applied on-the-fly, effectively improves performance in speaker verification tasks for these languages, achieving significant reductions in Equal Error Rate (EER) compared to traditional methods. This highlights the potential of SpecAugment as a simple yet powerful augmentation technique, particularly beneficial for low-resource language processing tasks.
• Relevance: The relevance of the article to the topic of low-resource language Automatic Speech Recognition (ASR) lies in its exploration of SpecAugment as a data augmentation technique for speaker verification tasks in languages like Cantonese and Tagalog, which are considered low-resource. By demonstrating the effectiveness of SpecAugment in improving performance in speaker verification tasks for these languages, the study showcases a potential solution to the challenges posed by limited data availability in low-resource language ASR. This highlights SpecAugment as a valuable tool for enhancing ASR systems' robustness and accuracy in under-resourced linguistic contexts.

Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., ... & Wu, Y. (2023). Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037.

• Summary: Google's Universal Speech Model aims to develop an ASR model that will be able to perform speech recognition on all languages of the world. This paper leverages large amounts of unlabelled speech and text data from YouTube to train a multilingual-encoder that can then be used in fine-tuning on very small amounts of labelled data. This allows them to outperform Whisper with significantly less labelled data, while also showing that this approach works positively for lower-resource languages.
• RQ: Can we leverage the large amounts of unlabelled speech data to perform massively multilingual ASR and speech translation?
• Hypothesis: By using a vast amount of unlabelled data, the encoder will learn speech representations that can be leveraged in fine-tuning and downstream tasks.
• Conclusion: Pre-training on unlabelled data is an effective way to improve multilingual performance while requiring much less labelled data.
• Critical observations: Although they keep mentioning that their performance is stellar on low-resource languages, no results were presented for these languages specifically. Most results are from multilingual datasets that might be imbalanced as well. Furthermore, the models and training data are not publicly available, making the research less approachable for improvements.
• Relevance: This paper is highly relevant for our theme as it aims to improve low-resource ASR through unlabelled data, which is an effective solution to the data scarcity problem.

Zhang, Y., Herygers, A., Patel, T., Yue, Z., & Scharenborg, O. (2023). Exploring data augmentation in bias mitigation against non-native-accented speech (arXiv:2312.15499). arXiv. http://arxiv.org/abs/2312.15499

• Summary: The study aimed to investigate the impact of data augmentation techniques on the performance of Flemish Automatic Speech Recognition (ASR) systems for both native Flemish speakers and those with non-native accents. Specifically, the research focused on addressing biases against non-native-accented Flemish speech. Various data augmentation methods were applied to augment the training data, and the performance of the ASR system was evaluated using both native and non-native speakers' speech samples. The results suggested that tailored data augmentation techniques can lead to improved ASR system performance for both native and non-native-accented Flemish speech. This finding highlights the potential of data augmentation in mitigating bias and enhancing the accuracy of ASR systems across diverse speaker demographics.
• RQ: What is the optimal type of data augmentation, in terms of reducing bias against non-native-accented Flemish in a Flemish ASR system, when applied to both native Flemish and non-native-accented Flemish?
• Hypothesis: Applying specific types of data augmentation techniques, tailored to address bias against non-native-accented Flemish speech, will lead to improved performance in a Flemish Automatic Speech Recognition (ASR) system for both native Flemish and non-native-accented Flemish speakers.
• Conclusion: The study concluded that employing tailored data augmentation techniques can significantly improve the performance of Flemish Automatic Speech Recognition (ASR) systems, particularly in mitigating biases against non-native-accented speech. By augmenting the training data with techniques specifically designed to address the characteristics of non-native accents, the ASR system demonstrated notable enhancements in accuracy for both native and non-native speakers. These findings underscore the importance of considering diversity in training data and utilizing appropriate augmentation strategies to enhance the robustness and inclusivity of ASR systems.
• Critical observations: The performance of Flemish Automatic Speech Recognition (ASR) systems can be significantly improved through the use of tailored data augmentation techniques. Specifically, augmenting the training data with methods designed to address the characteristics of non-native accents resulted in notable enhancements in accuracy for both native and non-native speakers. This observation highlights the importance of considering diversity in training data and employing appropriate augmentation strategies to enhance the inclusivity and robustness of ASR systems.
• Relevance: Low-resource languages often suffer from limited available data for training ASR systems, which can lead to poor performance, especially for speakers with non-native accents. This study demonstrates that tailored data augmentation techniques can substantially improve the accuracy of ASR systems, even in scenarios with limited training data.By addressing the challenges faced by speakers with non-native accents, the paper contributes valuable insights into how ASR technology can be adapted and optimized for low-resource languages. It underscores the importance of developing strategies that account for linguistic diversity and accent variations, ultimately making ASR systems more inclusive and effective in diverse linguistic contexts. Therefore, the findings of this study are highly relevant for researchers and practitioners working on ASR for low-resource languages, offering practical approaches to enhance system performance and usability in such settings.

Wang, H., Wang, S., Zhang, W. Q., & Bai, J. (2023). Distilxlsr: A light weight cross-lingual speech representation model. arXiv preprint arXiv:2306.01303.

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

N, K. D., Wang, P., & Bozza, B. (2021). Using Large Self-Supervised Models for Low-Resource Speech Recognition. Interspeech 2021, 2436–2440. https://doi.org/10.21437/Interspeech.2021-631

• Summary: This paper investigates the effectiveness of using large self-supervised pre-trained models (such as wav2vec 2.0) for low-resource speech recognition tasks. The authors conducted experiments on three Indian languages (Telugu, Tamil, and Gujarati), using different pre-trained models (monolingual English, multilingual) and compared different fine-tuning strategies (CTC, seq2seq, etc.).
• RQ: For low-resource speech recognition tasks, how effective are large self-supervised pre-trained models (such as wav2vec 2.0) compared to traditional supervised learning methods? For Indian languages, are cross-lingual multilingual pre-trained models or monolingual English pre-trained models more suitable? How do different fine-tuning strategies (CTC vs seq2seq) affect model performance? Additionally, how well do these pre-trained models generalize to seen and unseen languages?
• Hypothesis:
  1. Large self-supervised pre-trained models will outperform supervised learning models under low-resource conditions.
  2. Cross-lingual multilingual pre-trained models will perform better than monolingual English models on these Indian languages.
  3. Adopting the CTC fine-tuning strategy will achieve better performance than the seq2seq strategy.
• Conclusion:The multilingual pre-trained model XLSR outperformed the monolingual models on all three languages; for seen languages (like Tamil), the pre-trained model can approach the best performance with only 50% of the training data; the CTC fine-tuning framework performed better than the seq2seq framework, possibly due to the small amount of data; even smaller English pre-trained models showed decent transfer performance on Indian languages.
• Critical observations:The authors did not explain why the larger English pre-trained model underperformed compared to the smaller one, and analysis of the multilingual fine-tuning strategy was limited, only compared to the monolingual strategy. In addition, the impact of different pre-training corpora on model performance was not explored.
• Relevance:This work is important for low-resource speech recognition domains in developing countries. Leveraging large self-supervised pre-trained models can make full use of unlabeled data, alleviating the bottleneck of limited labeled data. This study provides an effective solution for low-resource speech recognition tasks.

Thomas, B., Kessler, S., & Karout, S. (2022). Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition (arXiv:2202.03218). arXiv. http://arxiv.org/abs/2202.03218

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Schultz, B.G., Tarigoppula, V.S.A., Noffs, G. et al. Automatic speech recognition in neurodegenerative disease. Int J Speech Technol 24, 771–779 (2021). https://doi-org.proxy-ub.rug.nl/10.1007/s10772-021-09836-w

• Summary: The paper evaluates the performance of three state-of-the-art automatic speech recognition (ASR) platforms (Amazon Web Services, Google Cloud, and IBM Watson) on speech from individuals with neurodegenerative diseases (multiple sclerosis and Friedreich's ataxia) and healthy controls.
• RQ: How well do commercial ASR systems perform on dysarthric speech from individuals with neurodegenerative diseases compared to healthy speech?
• Hypothesis: ASR accuracy will be lower for dysarthric speech from neurodegenerative disease groups compared to healthy controls, and accuracy will decline with increased disease severity and duration.
• Conclusion: ASR accuracy was significantly higher for healthy controls than clinical groups, and higher for multiple sclerosis compared to Friedreich's ataxia. Amazon Web Services and Google Cloud outperformed IBM Watson. Accuracy decreased with increased disease duration for Friedreich's ataxia but not multiple sclerosis. Age and sex did not significantly affect ASR accuracy.
• Critical observations:
  • ASR faces challenges in recognizing dysarthric speech from neurodegenerative diseases.
  • Accuracy declines as consecutive words increase, irrespective of speech impairment.
  • Severity of speech impairment, as indicated by disease type and duration, negatively impacts ASR accuracy.
• Relevance: The theme focuses on low-resource ASR for underrepresented languages. While this study does not directly address low-resource languages, it highlights the challenges ASR systems face in recognizing atypical speech patterns, which is relevant for low-resource languages with diverse speaker populations and dialects. Improving ASR performance on dysarthric speech could inform techniques for handling speech variability in low-resource settings.

Vásquez-Correa, J. C., Rios-Urrego, C. D., Arias-Vergara, T., Schuster, M., Rusz, J., Nöth, E., & Orozco-Arroyave, J. R. (2021). Transfer learning helps to improve the accuracy to classify patients with different speech disorders in different languages. Pattern Recognition Letters, 150, 272–279. https://doi.org/10.1016/j.patrec.2021.04.011

• Summary: The paper proposes using transfer learning with convolutional neural networks (CNNs) to classify pathological speech from patients with neurodegenerative disorders like Parkinson's disease (PD) and Huntington's disease (HD). Time-frequency representations of voice onset/offset segments are used as input to the CNNs. Two transfer learning scenarios are explored: 1) transferring a model trained on one language to classify patients speaking a different language, and 2) transferring a model trained on one disorder (e.g. PD) to classify patients with a different disorder (e.g. HD).
• RQ: Can transfer learning improve the accuracy of CNN models for classifying pathological speech across different languages and disorders?
• Hypothesis: Transferring knowledge from a base model trained on one language/disorder to a target model for a different language/disorder can improve classification accuracy when there is limited data for the target task.
• Conclusion: The results suggest transfer learning can improve target model accuracy, but only when the base model is sufficiently accurate. Transferring between similar tasks (e.g. different languages) works better than transferring between very different tasks (e.g. different disorders).
• Critical observations:
  • Accuracies ranged from 70-89% across languages without transfer learning
  • Transferring between languages improved accuracy in some cases (e.g. Spanish -> German improved over training on German alone)
  • Transferring between very different disorders like PD and HD did not improve over training directly on the target disorder
• Relevance: The paper does not directly address low-resource ASR, but instead focuses on pathological speech classification. However, some insights around transfer learning across languages could potentially be adapted to low-resource ASR scenarios.

Synthesis

In summary, these articles investigate various approaches to enhancing Automatic Speech Recognition (ASR) systems, particularly focusing on low-resource languages and accent variations. SpecAugment demonstrates effectiveness in speaker verification tasks across different languages, while Google USM explores leveraging unlabelled data for multilingual ASR. Additionally, data augmentation techniques are shown to mitigate biases against non-native accents in Flemish ASR systems. These findings collectively underscore the importance of robust and inclusive ASR technology for diverse linguistic contexts, prompting further exploration into tailored augmentation strategies and multilingual model development to address the challenges of low-resource languages and accent diversity. The combination of transfer learning and data augmentation have also shown potential for improving ASR performance when only limited data is available, by leveraging knowledge from higher-resource languages or domains.

Contributors

Contributors: Ömer Tarik, Xinyi Ma, Cantao Su, Page Ouyang, Weixi Lai, Xueying Liu

• Artice Wang et al., 2019: Xinyi Ma
• Article Google USM: Scaling automatic speech recognition beyond 100 languages: Ömer Tarik
• Article Zhang et al., 2023: Xinyi Ma
• Article Wang, H. et al., 2023: Page Ouyang
• Article N, K. D et al., 2021: Weixi Lai
• Article Thomas et al., 2022: Xueying Liu
• Article Automatic speech recognition in neurodegenerative disease, 2021: Cantao Su
• Article Transfer learning helps to improve the accuracy to classify patients with different speech disorders in different languages, 2021: Cantao Su
• Introduction: Ömer Tarik, Cantao Su
• Synthesis: Xinyi Ma, Cantao Su

Language-specific Text-To-Speech

Introduction

State-of-the-art Text-to-Speech systems have different performances based on the language they are developed for and trained on. We choose to focus on language-specific TTS and provide a review on state-of-the-art techniques to synthesise languages that are not English. This does not necessarily restrict to Low-Resourced Languages, but it will include them.

Article summaries

Do, P., Coler, M., Dijkstra, J., & Klabbers, E. (2021). A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages: Interspeech 2021. Proc. Interspeech 2021, 16–20. https://doi.org/10.21437/Interspeech.2021-1565

• Summary: The article provides an overview of strategies for text-to-speech (TTS) for low-resource langauges (LRLs), focusing on Multilingual Data strategies. More specifically, this article presents an evaluation of the results of the previous studies on LRLs TTS, an evaluation of the influence of data augmentation techniques employed on the performance of the models and the proposal of a new measure to evaluate the performance of multilingual vs. monolingual systems with different evaluation metrics, namely MultiLingual Model Effect (MLME). The performance of the strategies analysed is also checked by verifying how different factors influence it.
• RQ:
  1. Using the same limited amount of LRL data, how does the output quality of multilingual TTS models compare to that of monolingual models?
  2. What factors in the data augmentation strategy influence the effect of using multilingual TTS models on output quality, and to what extent do they affect it?
• Hypothesis: Looking at the correlations between data augmentation strategies and synthesized speech quality, tools that use multilingual data can be provided for future research in TTS for LRLs, especially regarding the efficiency of using such data.
• Conclusion: Multilingual approaches are more effective in training for LRLs. The factors that affect the performance are:
  • target language data ratio between corresponding multilingual and monolingual models;
  • target language data balance ratio over total training data
  • amount of target language data.
• Critical observations: The paper only focuses on multilingual data strategies, and justifies the choice by saying that multispeaker data are harder to collect for LRLs. Even though I understand the reasoning behind this, I believe this is not entirely true. On one hand, it is true indeed that it is harder to find many speakers for a LRLs, since oftentimes such languages are also minority languages. On the other hand, collecting multispeaker data means that each speaker can contribute with a very small amount of data and still get enough of them. This means that by adopting multispeaker TTS techniques, we don't need to record one speaker for a long time, but rather multiple speakers for a short time. This multi-speaker approach, I believe, could be used in combination with Transfer Learning to improve the results of LRLs TTS systems, even though this implies adding complezity to the pipeline.
• Relevance: The most relevant outcome of this study, especially for LRLs TTS, is that the language family is not relevant for the selection of the target-source language pair, no matter the architecture. In my opinion, the conclusions of this paper are also relevant for medium-resourced languages and in general for the synthesis of non-standard speech and for all the types of speech that are not widely covered by the research so far.

Staib, M., Teh, T. H., Torresquintero, A., Mohan, D. S. R., Foglianti, L., Lenain, R., & Gao, J. (2020). Phonological features for 0-shot multilingual speech synthesis. arXiv preprint arXiv:2008.04107.

• Summary: This article primarily aims to utilize a limited set of phonological features (PF), derived from the International Phonetic Alphabet (IPA), for achieving 0-shot speech synthesis and code-switching within a monolingual model. Specifically, the study selects Tacotron 2 as the baseline for comparison against methods of random initialization (RANDOM), manual mapping (MANUAL), and the PF-based approach (AUTO) proposed in this work. The conclusion drawn is that the speech generated using the AUTO method is more comprehensible.
• RQ: The research question of this paper explores whether phonological features (PF) can facilitate speech synthesis for languages not seen during training. Additionally, it examines whether PF can facilitate code-switched speech synthesis.
• Hypothesis: The hypothesis of the article is that phonological features (PFs), derived from the International Phonetic Alphabet (IPA), can enable 0-shot text-to-speech (TTS) synthesis and code-switching in languages that are not seen during training, even within monolingual models.
• Conclusion: The conclusion of the article is that by replacing the character input in Tacotron 2 with phonological features, a model topology can be created that is language-independent and allows for the automatic approximation of sounds unseen in training. The study shows that phonological features (PFs) can not only facilitate zero-shot speech synthesis in untrained languages within a small multilingual or even a monolingual model but also facilitate the synthesis of sounds that completely unseen in training. This suggests potential applications in code-switching and TTS for low-resource languages.
• Critical observations: This article mainly addresses the problem of 0-shot speech synthesis and code switching by using phonological features (PF). The significant advantage of this method is that it can reduce the amount of data for training multi-language speech synthesis models, and it is very helpful in low-resource languages ​​and code-switched TTS. But PF may not capture all the differences of a language, especially for those with unique phonetic and phonological features, and the selected PFs might not adequately represent these languages. In addition, this article focuses more on generating understandable speech and may ignore the importance of features such as prosody.
• Relevance: This paper is mainly related to the fileds of cross-language speech synthesis and code switching speech synthesis. Some other studies have also proposed to find a unified representation (such as Unicode) to replace phoneme or text to achieve cross-language synthesis, but the PF proposed in this paper may be A better choice because these features retain speech features to a certain extent and help the model learn better.

Do, P., Coler, M., Dijkstra, J., & Klabbers, E. (2023). Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection. arXiv preprint arXiv:2306.12040.

• Summary: This paper compares two methods in TTS for low-resource languages: PHOIBLE-based phone mapping and phonological features input. Various languages are tested to see how these methods work across different languages. The findings show that both methods improve speech quality, with phonological features performing better. The study also examines two criteria for choosing source languages: Angular Similarity of Phone Frequencies (ASPF) and language family tree distance. ASPF is found effective, especially with phone-based input, while the language distance criterion does not yield expected results.
• RQ: The paper aims to explore how to most effectively deal with the input mismatch between languages and how to select the best source language to improve output quality in TTS for low-resource languages.
• Hypothesis:
  1. Transfer learning using PHOIBLE-based phone mapping and phonological feature inputs can improve TTS output quality for low-resource languages.
  2. Angular Similarity of Phone Frequencies (ASPF) is an effective criterion for selecting source languages, more so than traditional broad language family classification.
• Conclusion:
  1. Both phone mapping and feature inputs can enhance output quality, with feature inputs showing better performance, although the effectiveness depends on the specific language pairing.
  2. ASPF is effective in selecting source languages, especially when using label-based phone inputs, while the distance based on the language family tree does not work as expected.
• Critical observations:
  1. Although ASPF is effective in some cases, its effectiveness is not universal across all language combinations, indicating the need for further research to understand influencing factors.
  2. The unexpected results with the language family tree distance suggest that there might be unidentified factors at play, necessitating further investigation.
• Relevance: This research is significant for the development of TTS technology for low-resource languages, especially in offering new insights into source language selection and handling input mismatches between languages. Moreover, the proposed methods are important for the multilingual applicability and scalability of speech technologies.

Wells D, Richmond K. Cross-lingual transfer of phonological features for low-resource speech synthesis[C]//Proceedings of the 11th Speech Synthesis Workshop, Budapest, Hungary. 2021: 160-165.

• Summary: In this paper, researchers compare two methods: fine-tuning phonemic representations and using phonological features. They used SPE-style phonological features, offering a binary representation of phonemes, which helps describe and analyze speech patterns in English and German. The study discovers that even with limited target language data, fine-tuning can generate speech comparable to models trained from scratch. Using phonological features slightly improves naturalness ratings compared to using phonemes alone. These findings highlight the practical benefits of phonological features in improving TTS output quality across languages.
• RQ: Does the use of different input representations (phonemes and phonological features) affect the naturalness of synthesized speech in text-to-speech synthesis using cross-lingual transfer learning?
• Hypothesis: In cross-lingual transfer learning for text-to-speech synthesis, the use of different input representations (phonemes and phonological features) affects the naturalness of synthesized speech.
• Conclusion: The study confirmed the effectiveness of cross-lingual fine-tuning for training synthetic voices with limited target language data. Phonological features were found to offer practical benefits over phonemes in terms of parameter sharing during transfer learning.
• Critical observations: There was a slight improvement in naturalness ratings when using PFs over phonemes. Future research may explore multilingual grapheme-to-phoneme systems and utilize additional linguistic resources to enhance low-resource pipelines for text-to-speech synthesis
• Relevance: Phonological features were found to offer practical benefits over phonemes in terms of parameter sharing during transfer learning, which can be applied greatly in LRLs TTS.

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

• Article Do, et al. (2021) 'A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages': Alice Vanni
• Article Jones et al. 2023: YOUR NAME
• Article Staib et al. (2020) 'Phonological features for 0-shot multilingual speech synthesis': Wang Yinqiu
• Article Do, et al. (2023) 'Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection': Annie Zhou
• Article Wells D, et al. (2021) 'Richmond K. Cross-lingual transfer of phonological features for low-resource speech synthesis': Ding
• Introduction: All
• Synthesis: All

Theme: Non-Language-specific Text-To-Speech

Introduction

TTS systems have significantly advanced over time, achieving remarkable intelligibility and near-human naturalness in synthetic voices through deep learning advancements. However, the naturalness of synthetic voice remains limited to sentences, and lacks the expressivity found in human conversation such as appropriate emotion, prosody and style. Despite these limitations, natural TTS, particularly expressive speech synthesis, plays a crucial role in achieving human-like speech and enhancing the engagement of synthesized speech. Moreover, it facilitates the broader adoption of TTS technology across various domains within the field of speech technology. In this context, our group focuses on the theme of TTS naturalness with two interconnected subthemes: exploring advanced models and relevant theories. By addressing these subthemes, we aim to provide a comprehensive overview of the current state-of-the-art in TTS naturalness.

Article summaries

Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., ... & Zhao, Z. (2023, July). Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning (pp. 13916-13932). PMLR.

• Summary: The article introduces "Make-An-Audio," a system utilizing a prompt-enhanced diffusion model for TTS generation, aiming to improve the naturalness and expressiveness of synthesized audio.
• RQ: How does the model improve the naturalness of TTS?
• Hypothesis: By introducing pseudo prompt enhancement and spectrogram autoencoders, the model can effectively utilize unsupervised language-free data and higher-level semantic understanding to enhance the naturalness and expressiveness of speech synthesis.
• Conclusion: "Make-An-Audio" successfully enhances the naturalness and expressiveness of speech synthesis, achieving state-of-the-art performance in evaluations.
• Critical observations: The performance of "Make-An-Audio" is still partly dependent on extensive data and complex model training. In addition, there is still space for improvement in expressing the emotions and rhythms of human conversations.
• Relevance: The "Make-An-Audio" system presented in the paper offers an effective solution to the limitations in naturalness and expressiveness currently faced by TTS.

APA Citation of NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality

• Summary: NaturalSpeech proposes a system for converting text to speech (TTS) that achieves human-level quality. It leverages a variational autoencoder (VAE) to bridge the gap between text and speech waveforms.
• RQ (Research Question): Can a TTS system achieve speech quality indistinguishable from humans?
• Hypothesis: By incorporating a VAE and specific techniques to improve the model's understanding of text and speech features, NaturalSpeech can generate speech indistinguishable from humans.
• Conclusion: The paper argues that NaturalSpeech achieves human-level speech quality based on statistical measures (MOS and CMOS) in human evaluations.
• Critical Observations: The evaluation relies on subjective human ratings, which might be influenced by factors beyond speech quality.The research focuses on a single benchmark dataset, limiting generalizability.The paper doesn't explore how NaturalSpeech performs on diverse speaking styles or accents.

• Relevance: This is related to my study because it provides a definition of human-level quality, and this particular model has achieved the highest Mean Opinion Score (MOS) recorded thus far. Hence, I am considering using this model as a basis for my study.

Noufi, C., May, L., & Berger, J. (2023). Context, Perception, Production: A Model of Vocal Persona. PsyArXiv. July, 28.

• Summary: This article introduces a contextualized production-perception model of vocal persona, developed through qualitative analysis of interviews with voice and performance experts. It emphasizes the influence of context on an individual's vocal expression, reflecting the intricacies of human communication.
• RQ: What is the relationship between context, vocal expression, and identity?
• Hypothesis: It is a qualititative study and does not hve a formulated hypothesis. Instead of attempting to falsify a hypothesis as in most quantitative studies, it explores answers to the research question through thematic analysis.
• Conclusion: Speakers actively select different vocal personas and adjust relevant vocal expressions in response to the surrounding context, facilitating a transition in the perception of persona.
• Critical observations: The proposal of the vocal persona model and general conclusions are based on interviews with 21 voice and performance experts, which may have limitations in terms of subjective bias and generalizability beyond this specific context.
• Relevance: This study underscores the necessity for speakers to adapt their speaking styles to accommodate different social contexts, highlighting the significance of context in vocal expression. It proposes the incorporation of vocal persona into expressive vocal synthesis with a three-spoke model and a framework for persona-guided vocalization, enriching the framework of TTS naturalness and expressiveness.

Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity

• Summary: This article introduces HiFi-GAN, a model that can efficiently synthesize high-quality speech audio. HiFi-GAN consists of a generator and two discriminators: multi-scale discriminator and multi-period discriminator. Improve training stability and model performance by adversarially training the generator and discriminator and using two additional loss functions.
• RQ:Can HiFi-GAN effectively synthesize high-quality speech audio with computational efficiency comparable to human-level synthesis, while also demonstrating generalization across speakers and adaptability to various configurations?
• Hypothesis:By leveraging the characteristic patterns of speech audio and designing a discriminator to capture these patterns effectively, it is possible to develop a speech synthesis model, HiFi-GAN, that outperforms existing models in terms of synthesis quality and speed.
• Conclusion:HiFi-GAN significantly advances speech synthesis by efficiently generating high-quality audio, surpassing existing models in both synthesis quality and speed. By leveraging speech audio patterns and a carefully designed discriminator, this model demonstrates robustness across various scenarios, including unseen speakers and noisy inputs, while offering potential for on-device natural speech synthesis with low latency and memory requirements. Additionally, the flexibility of generator configurations enhances adaptability without the need for extensive hyper-parameter search.
• Critical observations:Due to the wide application of HiFi-GAN technology in the field of speech synthesis, there may be some ethical or social impacts, including concerns related to voice cloning, privacy and false information.
• Relevance:This paper is closely related to the topic of non-language-specific text-to-speech, as it demonstrates a breakthrough in HiFi-GAN models in synthesizing high-quality speech, with generalization capabilities, and the ability to handle inputs of different languages and speaking styles.

Jan Vainer & Ondˇ rej Dušek(2020). SpeedySpeech: Efficient Neural Speech Synthesis

• Summary: This paper introduces a novel student-teacher network architecture called "SpeedySpeech" for fast and high-quality neural speech synthesis. The system is designed to enable faster-than-real-time speech synthesis while requiring minimal computing resources, and deliver audio quality that is superior to existing models such as the Tacotron 2. The model uses the teacher network for duration extraction, the student network for spectrogram synthesis, and combines it with the MelGAN vocoder to output high-quality audio. The training process is efficient and can be completed in less than 40 hours on a single 8GB GPU.
• RQ: How can we develop a neural speech synthesis system that does not require extensive computing resources while maintaining fast training times, fast inference, and high-quality audio output?
• Hypothesis: Assuming a student-teacher network architecture with simplified convolutional blocks and only a single attention layer in the teacher model, it is possible to surpass existing models in terms of training efficiency and audio quality while maintaining fast inference.
• Conclusion: The proposed SpeedySpeech model successfully achieves its goals by demonstrating that self-attention layers are not necessary for high-quality audio generation and that simpler, fully convolutional methods enable a more efficient training process and faster synthesis. The model's speech quality score is significantly higher than Tacotron 2, and it can be trained efficiently on a single GPU and even run in real time on the CPU.
• Critical observations: The article proposes ways to address the trade-off between training efficiency and audio quality in neural speech synthesis. By using only a single attention layer in the teacher model and eliminating sequence generation in the student network, the authors achieve important simplifications that increase model efficiency. In the model evaluation, the authors comprehensively considered objective indicators (such as MAE, SSIM) and subjective listening tests to provide a comprehensive assessment of model performance.
• Relevance: This speech synthesis model has applications in many fields, including virtual assistants, machine translation, etc. The SpeedySpeech model can synthesize speech in real time on moderate hardware, making it particularly suitable for deployment in resource-constrained environments. Additionally, the focus on efficiency and quality sets new benchmarks for future research and development in this area.

Peiró-Lilja, A., & Farrús, M. (2020). Naturalness Enhancement with Linguistic Information in End-to-End TTS Using Unsupervised Parallel Encoding. Interspeech 2020, 3994–3998. https://doi.org/10.21437/Interspeech.2020-1788

• Summary: The paper explores enhancing the naturalness of synthesized speech in E2E-TTS systems by incorporating linguistic features like POS tags and punctuation into the Tacotron 2 model, aiming to improve prosody to resemble human speech more closely.
• RQ: How can linguistic information be integrated into the Tacotron 2 system to improve the naturalness of synthesized speech prosody?
• Hypothesis: The hypothesis is that by embedding POS tags and punctuation locations as additional linguistic features into the Tacotron 2 system, the synthesized speech will exhibit improved naturalness and prosody, making it more similar to human speech.
• Conclusion: The study concludes that the incorporation of linguistic features through a parallel encoder significantly improves the naturalness of synthesized speech. The authors proposed two different architectures for the parallel encoder: one based on convolutional and recurrent layers (2DConv+BiLSTM) and another composed of bidirectional recurrent and linear layers (BiGRU+Linear). Both architectures aimed to process the binary matrix representing POS tags and punctuation locations. The results from objective tests and perceptual evaluations indicated that the model with the 2DConv+BiLSTM parallel encoder performed the best in terms of naturalness, as it more closely matched human pitch contours and overall speech quality.
• Critical observations: Critically, the paper notes that while both parallel encoder architectures showed improvements over the Tacotron 2 baseline, the 2DConv+BiLSTM version provided better results in terms of naturalness. However, it also introduced a slight increase in Mel Cepstral Distortion (MCD), suggesting a trade-off between naturalness and certain acoustic quality metrics. The BiGRU+Linear model, despite being lighter and faster, underperformed in perceptual tests, possibly due to its reduced complexity and higher cepstral distortion.
• Relevance: The findings of this research are relevant for the development of more natural and human-like E2E-TTS systems, which have applications in various domains such as automatic dialogue systems, storytelling, and voice assistants. By enhancing the prosody of synthesized speech, these systems can provide more engaging and realistic interactions, improving user experience and accessibility. Furthermore, the study contributes to the broader field of speech synthesis by demonstrating the potential of unsupervised parallel encoding of linguistic features to improve speech naturalness.

Cai, X., Dai, D., Wu, Z., Li, X., Li, J., & Meng, H. (2021). Emotion Controllable Speech Synthesis Using Emotion-Unlabeled Dataset with the Assistance of Cross-Domain Speech Emotion Recognition. ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5734–5738. https://doi.org/10.1109/ICASSP39728.2021.9413907

• Summary: This article proposes an approach for emotional TTS synthesis on a dataset without emotion labels, using a cross-domain speech emotion recognition model and an emotional TTS model, hoping to express similar results in specific emotional expressiveness and speech quality compared to models with emotion labels.
• RQ: Can we use the achievements and features of SER to solce the problem of lack of emotion-annotated dataset for emotional TTS?
• Hypothesis: By performing mean opinion score evaluations and emotion recognition perception evaluation in 4 emotion categories and 2 polarities of emotion dimensions, our GST-based model can generate speech with expected emotions, while trained on a fully emotion-unlabeled dataset.
• Conclusion: Through comparing their created cross-domain model with a baseline model, they found that both their 4-categorical model and 2-dimensional model almost achieve as good speech quality as the baseline system, with higher p-values than the significance level of 0.05, indicating no significant differences. Furthermore, they found that both categorical models, one trained on the utterances with highest posterior (top-K scheme) and one trained on the full set of audio, revealed an overall higher accuracy than the baseline model, at 78.75% and 49.25%, respectfully, compared to the baseline 36.75%, which indicated their cross-domain model and top-K scheme were effective in emotional expressiveness.
• Critical observations: The choice to use a top-K scheme, as described earlier, is interesting to offset the number of mispredictions made by the SER model, as the SER model is far less reliable that humans. By choosing to use the more reliable audio set, there could be an argument that their choice could inflate their results. However, taking this into account, they did in fact train the model on the full, unaltered, set of audio and still returned a higher accuracy than the baseline model. The fact that this shows promise in returning accurte and sufficient quality in emotional speech synthesis with unlabeled emotion datasets brings promise to a possible increase in speed and efficiency for training other models.
• Relevance: The proposed approach, in the authors' words, greatly reduces the threshold of emotional synthesis in regard to amotion-annotated data, reducing the time, cost, and relevant quality of the speech data needed for emotional TTS systems.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Huang et al. 2023: Yilan Wei
• Article Context, Perception, Production: A Model of Vocal Persona: Chenyi Lin
• Article NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality: Yi Lei
• Article HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis: Yanhua, Liao
• Article Naturalness Enhancement with Linguistic Information in End-to-End TTS Using Unsupervised Parallel Encoding: Jingxuan Yue
• Article Emotion Controllable Speech Synthesis Using Emotion-Unlabeled Dataset with the Assistance of Cross-Domain Speech Emotion Recognition: Jocomin Galarneau
• Article XXX: YOUR NAME
• Introduction: Chenyi Lin
• Synthesis: All

Theme: ASR

Introduction

The rapid evolution of Automatic Speech Recognition (ASR) technology has been a cornerstone in advancing how humans interact with machines, propelling us towards more seamless and intuitive communication avenues. The focus on ASR technology underscores its pivotal role across a myriad of applications, from enhancing accessibility and providing robust customer support solutions to creating immersive interactive entertainment experiences. Among the most intriguing challenges in this domain is the recognition and interpretation of complex human sentiments such as sarcasm and humor. These nuanced forms of expression, deeply embedded in human language, present unique challenges for ASR systems due to their reliance on contextual cues, background knowledge, and the subtle modulations in tone that conventional speech recognition systems often miss. Our exploration is driven by the imperative to bridge this gap, aiming to refine ASR technology's ability to discern and process these complex sentiments.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Wang S, Yang C H H, Wu J, et al. Can whisper perform speech-based in-context learning[J]. arXiv preprint arXiv:2309.07081, 2023.

• Summary: The study investigates Whisper ASR models' in-context learning capabilities and proposes a novel SICL method for test-time adaptation without gradient descent, achieving significant WER reductions.
• RQ: The research explores whether Whisper models can perform speech-based in-context learning and how to leverage in-context examples for test-time adaptation efficiently.
• Hypothesis: The hypothesis is that Whisper models can adapt at test time using SICL with context examples from specific dialects or speakers.
• Conclusion: SICL significantly improves ASR performance for Chinese dialects without gradient descent, with k-NN enhancing SICL's efficiency.
• Critical observations: Correct LID settings and k-NN example selection improve Whisper's inference, with language-level adaptation outperforming speaker-level adaptation.
• Relevance: The study is relevant for understanding and enhancing the application of large pre-trained models in automatic speech recognition and dialect adaptation.

Sungjoo Ahn and Hanseok Ko. “Background Noise Reduction via Dual-Channel Scheme for Speech Recognition in Vehicular Environment.” IEEE Transactions on Consumer Electronics 51, no. 1 (February 2005): 22–27. https://doi.org/10.1109/TCE.2005.1405694.

• Summary: The paper proposes a dual-channel noise reduction method aimed at enhancing speech recognition systems within vehicular environments, characterized by significant noise challenges. The authors argue that existing single-channel methods fall short in effectively improving speech recognition performance due to inherent noise complexities in vehicles. The proposed method leverages a high-pass filter combined with an eigen-decomposition front-end processing technique, tested against real multi-channel vehicular corpus. Experimental results indicated a notable improvement in speech recognition performance using various microphone arrangements, showcasing the superiority of the dual-channel approach over traditional single-channel methods.
• RQ: How can the performance of speech recognition systems in vehicular environments be improved through a dual-channel noise reduction scheme?
• Hypothesis: The paper hypothesizes that employing a dual-channel noise reduction scheme, which integrates a high-pass filter with eigen-decomposition front-end processing, can significantly enhance speech recognition performance in noisy vehicular environments by effectively distinguishing speech from background noise.
• Conclusion: Authors concluded that their dual-channel noise reduction method, especially when augmented with a high-pass filter and enhanced eigen-decomposition processing, substantially improves speech recognition accuracy in vehicular settings. The method outperformed standard single-channel noise reduction approaches and showed considerable promise in overcoming the challenges posed by vehicular background noise, thereby validating the hypothesis.
• Critical observations: The study successfully demonstrates the effectiveness of a dual-channel approach in a challenging noise environment. However, the practical deployment of such systems, including the economic implications and the adaptability across different vehicle models and noise conditions, remains less explored. Additionally, while the study marks a significant improvement over existing methods, the scalability of this approach in terms of computational demand and real-time processing capabilities could benefit from further investigation.
• Relevance: This thesis is relevant to the topics in enhancing speech recognition technology area. The innovative approach of combining a dual-channel noise reduction scheme with a high-pass filter and eigen-decomposition method provides a substantial leap forward in developing more reliable and efficient speech recognition systems.

Zhang, Wangyou, and Yanmin Qian. “Weakly-Supervised Speech Pre-Training: A Case Study on Target Speech Recognition.” arXiv, June 29, 2023. http://arxiv.org/abs/2305.16286.

• Summary: This study introduces a new way to teach computers to understand speech by focusing on one person's voice in a noisy place, like when many people talk at once. This method, called TS-HuBERT, uses extra information about the speaker's voice to improve speech recognition, especially in challenging situations with lots of background noise. Tests showed that TS-HuBERT does a better job than other similar methods, making it a promising approach for better understanding speech in noisy environments.
• RQ: Can we use extra information about who is speaking to help computers better recognize speech in noisy settings?
• Hypothesis: By using additional information about the speaker, the TS-HuBERT method can focus on the target speaker's voice more effectively, even when other voices or noises are present.
• Conclusion: TS-HuBERT improves speech recognition by focusing on the target speaker's voice, outperforming other current methods. This approach is particularly useful for recognizing speech in noisy places where many people are talking at once.
• Critical observations:
  • TS-HuBERT can be adjusted to different speech recognition tasks, showing its versatility.
  • Although it needs extra information about the speaker's voice, this method greatly enhances the computer's ability to focus on and understand the target speaker in noisy situations.
  • There is still room for improvement, especially in very noisy environments, indicating potential areas for future research.
• Relevance: This study is directly relevant to the topic to help computers understand speech better in challenging environments, like when many people are talking at the same time. By focusing on a specific speaker's voice, TS-HuBERT could make speech recognition technology more effective in real-world situations.

Bae, S., Kim, J.-W., Cho, W.-Y., Baek, H., Son, S., Lee, B., Ha, C., Tae, K., Kim, S., & Yun, S.-Y. (2023). Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification. Retrieved from https://arxiv.org/abs/2305.14032v4

Summary: The study introduces a novel approach for respiratory sound classification, leveraging a pretrained Audio Spectrogram Transformer (AST) model, alongside a new Patch-Mix augmentation technique and Patch-Mix Contrastive Learning. These methods are designed to address the challenges of medical data scarcity and enhance model performance on the ICBHI dataset. The approach sets a new state-of-the-art performance benchmark, improving the classification Score by 4.08% over previous methods.

• RQ: Can a pretrained Audio Spectrogram Transformer (AST) model, combined with Patch-Mix augmentation and Patch-Mix Contrastive Learning, effectively improve respiratory sound classification, especially in the context of the ICBHI dataset?
• Hypothesis: The hypothesis posits that leveraging a pretrained AST model, which has been trained on large-scale visual and audio datasets, can be effectively generalized to respiratory sound classification tasks. Additionally, it suggests that the introduction of Patch-Mix augmentation and Patch-Mix Contrastive Learning can further enhance model performance by addressing the scarcity of medical data and the challenges of leveraging such data for deep learning models.
• Conclusion: The study concludes that the proposed approach, combining a pretrained AST model with Patch-Mix augmentation and Patch-Mix Contrastive Learning, significantly enhances respiratory sound classification. This method achieved state-of-the-art performance on the ICBHI dataset, demonstrating the effectiveness of the proposed techniques in improving classification accuracy in the face of limited medical data availability and complex data characteristics.
• Critical observations:
  • Pre-training on both visual and audio domains using the AST model shows substantial improvements in generalizing to respiratory sound classification tasks.
  • The Patch-Mix augmentation technique, which randomly mixes patches between different samples, and the Patch-Mix Contrastive Learning method, which distinguishes mixed representations in the latent space, effectively mitigate the overfitting issue and enhance model robustness.
  • The study's methodology offers a significant performance increase, demonstrating the potential of attention-based models and contrastive learning in medical sound classification.
• Relevance: This research holds relevance to Automatic Speech Recognition (ASR) by showcasing the utility of attention-based models like the AST in capturing long-range dependencies in audio data. The techniques developed for respiratory sound classification, particularly the effective use of pretrained models and innovative augmentation strategies, can inform similar challenges in ASR, including dealing with limited training data and enhancing model generalization across diverse audio inputs.

Gairola1, S., Tom, F., Kwatra1, N., & Jain1, M. (2021). RESPIRENET: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting. Retrieved from https://arxiv.org/abs/2011.00196v2

• Summary: The study introduces RespireNet, a CNN-based model for classifying respiratory sounds, particularly focusing on addressing the challenge posed by the small size of the largest available respiratory dataset, ICBHI, which consists of only 6,898 breathing cycles. The study proposes a suite of novel techniques including device-specific fine-tuning, concatenation-based augmentation, blank region clipping, and smart padding to efficiently utilize this small dataset. Extensive evaluation on the ICBHI dataset demonstrates significant improvements over state-of-the-art results for 4-class classification by 2.2%.
• RQ: Can a simple CNN-based model, when combined with specific data utilization techniques, accurately classify respiratory sounds from a limited-sized dataset, overcoming the challenges of data scarcity and variability?
• Hypothesis: The study hypothesizes that even with a small dataset, a simple network architecture, if supplemented with innovative techniques for data utilization and augmentation, can accurately classify respiratory sounds. These techniques include addressing dataset characteristics such as device variability, class imbalance, and varying audio lengths that traditionally inhibit effective DNN training.
• Conclusion: RespireNet, along with the proposed data utilization techniques, significantly improves the accuracy of respiratory sound classification, achieving new state-of-the-art performance on the ICBHI dataset for both 2-class and 4-class classification tasks. The study concludes that focusing on efficient data utilization and addressing specific dataset characteristics can compensate for the limitations posed by small-sized datasets.
• Critical observations:
  1. Transfer learning from pre-trained ImageNet models proves beneficial, suggesting that even unrelated domain knowledge can improve model performance.
  2. Concatenation-based augmentation effectively addresses class imbalance, significantly improving classification of underrepresented classes.
  3. Device-specific fine-tuning is essential for generalizing across different recording devices, highlighting the impact of hardware variability on model performance.
  4. Techniques like smart padding and blank region clipping are crucial for dealing with variable-length audio samples and irrelevant frequency regions, respectively, ensuring the model focuses on relevant features.
• Relevance: The challenges and solutions presented in this study have direct implications for ASR, especially in scenarios where data is scarce or highly variable. Techniques such as smart data augmentation, device-specific adjustments, and focusing on relevant audio features can be applied to improve ASR systems' robustness and accuracy in diverse conditions. Furthermore, the emphasis on efficient data utilization and simple model architectures can inspire similar approaches in ASR research to overcome data-related limitations.

Yang, R., Lv, K., Huang, Y., Sun, M., Li, J., & Yang, J. (2023). Respiratory Sound Classification by Applying Deep Neural Network with a Blocking Variable. Applied Sciences, 13(6956). https://doi.org/10.3390/app13126956

• Summary: The paper introduces a deep neural network named Blnet for classifying respiratory sounds, incorporating features from ResNet, GoogleNet, and self-attention mechanisms to tackle the non-IID (not independently and identically distributed) data problem and imbalanced data issues. The model demonstrated improved performance on the ICBHI 2017 respiratory sound database, showcasing a significant advancement in sensitivity and specificity rates over existing methods.
• RQ: How can a deep neural network be optimized for classifying respiratory sounds to facilitate the early detection of respiratory diseases, considering challenges such as non-IID data and imbalanced datasets?
• Hypothesis: The integration of ResNet, GoogleNet, and self-attention mechanisms into a deep neural network, alongside a two-stage training process and mix-up data augmentation within clusters, can significantly improve the classification accuracy of respiratory sounds, even with imbalanced and non-IID data challenges.
• Conclusion: The Blnet model successfully addressed the challenges of non-IID and imbalanced datasets in respiratory sound classification, achieving a 4.22% improvement in average score and a 12.61% improvement in sensitivity over state-of-the-art results. This performance enhancement underscores the efficacy of the proposed network architecture and training strategies.
• Critical observations:
  • The two-stage training process and the introduction of a blocking variable proved effective in managing non-IID data, suggesting the importance of considering data distribution in deep learning models.
  • Mix-up data augmentation within clusters and the use of multiple input transformations (STFT and WT) were critical in addressing data imbalance and enhancing model robustness.
  • The self-attention mechanism played a key role in capturing global dependencies within the data, improving the model's feature extraction capabilities.
  • Simplifying the loss function to handle a four-class classification task as two independent binary classification tasks was found to enhance training effectiveness.
• Relevance: The techniques and findings of this study have direct implications for ASR systems, particularly in enhancing model performance with non-IID and imbalanced datasets. The methods for improving feature extraction and classification in the context of respiratory sound analysis can inform approaches to noise reduction, signal processing, and robust model training in ASR technologies. Furthermore, the attention mechanisms and data augmentation strategies could be adapted to improve ASR systems' ability to deal with diverse and challenging acoustic environments.

Zhou, Rui, Xian Li, Ying Fang, and Xiaofei Li. “Mel-FullSubNet: Mel-Spectrogram Enhancement for Improving Both Speech Quality and ASR.” arXiv, February 21, 2024. http://arxiv.org/abs/2402.13511.

• Summary: This paper introduces Mel-FullSubNet, a network designed for enhancing speech quality and automatic speech recognition (ASR) performance. It focuses on improving both the clarity of speech and its recognizability by machines in noisy conditions. The technique enhances Mel-spectrograms of speech, which can then be used directly for ASR or converted back into speech waveforms using a neural vocoder. The method combines full-band and sub-band network processing, proving to be more effective for ASR and speech quality enhancement compared to previous approaches.
• RQ: Can Mel-spectrogram enhancement via Mel-FullSubNet improve both speech quality and automatic speech recognition performance in noisy conditions?
• Hypothesis: By enhancing Mel-spectrograms using the Mel-FullSubNet, which combines full-band and sub-band processing, both speech quality and ASR performance can be significantly improved in noisy environments.
• Conclusion: Mel-FullSubNet successfully enhances speech quality and ASR performance, outperforming several existing methods. It shows particular strength in providing cleaner speech signals and more accurate ASR results by focusing on Mel-spectrogram enhancement and efficiently leveraging neural vocoders for waveform generation.
• Critical observations:
  • Mel-FullSubNet demonstrates superior generalization to unseen data and environments, a critical advantage for real-world applications.
  • The method's efficacy is underscored by its performance on various datasets, indicating its robustness and adaptability.
  • While Mel-FullSubNet requires more computational resources due to its neural vocoder component, its efficiency and output quality justify the additional cost.
• Relevance:This study is directly relevant to the topic to the challenge of enhancing speech recognition systems in noisy conditions, a common problem in real-world applications. By focusing on Mel-spectrogram enhancement, Mel-FullSubNet provides a novel approach that benefits both speech clarity and ASR accuracy, making it a valuable reference for further research in speech processing technology.

Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., & Poria, S. (2019). Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). arXiv:1906.01815v1.

• Summary: The paper introduces a novel approach to sarcasm detection by leveraging multimodal data. Recognizing that sarcasm often involves incongruities not just in text but also in vocal tone and facial expressions, the authors propose the first dataset, MUStARD, for sarcasm detection using audio-visual cues alongside textual data. This dataset, compiled from popular TV shows, is annotated for sarcasm, aiming to facilitate the development of models that can better understand sarcasm through the integration of multiple modes of communication.
• RQ: How can incorporating multimodal cues (textual, audio, and visual) improve the automatic classification of sarcasm compared to relying on textual data alone?
• Hypothesis: The paper hypothesizes that the inclusion of multimodal information (audio and visual cues, along with textual data) can significantly enhance the performance of sarcasm detection models, reducing the relative error rate by up to 12.9% in F-score when compared to models that use only individual modalities.
• Conclusion: The research demonstrates that multimodal models significantly outperform unimodal variants in sarcasm detection, with a notable reduction in error rate. The findings underscore the importance of considering multiple communication cues, beyond just text, for effectively identifying sarcasm. The MUStARD dataset is also introduced as a valuable resource for future research in multimodal sarcasm detection.
• Critical Observations:

1. Sarcasm detection benefits from multimodal analysis, including textual, audio, and visual data, highlighting the complex nature of sarcasm as a communicative act that often relies on the interplay of various signals.
2. The MUStARD dataset fills a critical gap in research resources, providing a foundation for exploring how different modalities contribute to the understanding of sarcasm.
3. The study's methodology, focusing on a balanced dataset and robust multimodal feature extraction techniques, sets a precedent for future work in this area.

• Relevance: This research is highly relevant to my thesis topic. It pushes the boundaries of sarcasm detection by moving beyond text analysis to include audio and visual cues, offering insights into more holistic approaches to understanding human communication. The findings and the MUStARD dataset can significantly impact the development of more nuanced and effective computational models for detecting sarcasm and other complex emotional or figurative language use cases.

Zhang, Yazhou, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. “CMMA: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations,” n.d.

• Summary: This study introduces the CMMA dataset for benchmarking multi-affection detection in Chinese multi-modal conversations, focusing on sentiment, emotion, sarcasm, and humor. The dataset comprises annotations from a variety of TV series to reflect diverse affective expressions and supports both single-task and multi-task learning paradigms for affective computing research.
• RQ: How multi-modal cues and conversational context influence the detection of multiple affects, including sentiment, emotion, sarcasm, and humor, in Chinese multi-party conversations?
• Hypothesis: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations" likely centers on the premise that incorporating multi-modal data (text, video, audio) and conversational context significantly improves the accuracy and effectiveness of detecting multiple affects (sentiment, emotion, sarcasm, humor) in multi-party conversations. The study posits that the interplay between different modalities and the contextual understanding of conversations enhances the model's ability to interpret complex human affective expressions.
• Conclusion: The findings demonstrate that conversational context and multi-modal data significantly enhance affect detection tasks. The study also highlights the importance of multi-affect annotation for understanding complex human communications, suggesting the CMMA dataset as a valuable resource for future affective computing research.
• Critical observations: While the dataset offers comprehensive insights into multi-affect detection, its focus on Chinese TV series may limit its applicability across different linguistic and cultural contexts. Additionally, the inherent subjectivity of affect annotation poses challenges to achieving unbiased affect detection.
• Relevance: This study is pertinent to my thesis as it provides an opportunity to delve into how various feature fusion methods impact the accuracy of sarcasm recognition in Mandarin using multimodal data. Additionally, the CMMA dataset is highly beneficial to my research because it is among the few Chinese datasets that include labels for sarcasm, offering a valuable resource for studying sarcasm recognition within Mandarin-specific contexts using multimodal information.

Patel, T., & Scharenborg, O. (2024). Improving End-to-End Models for Children’s Speech Recognition. Applied Sciences, 14(6), 2353.

• Summary: Children’s Speech Recognition (CSR) is challenging due to variable speech patterns and limited annotated data. We aim to enhance CSR when no child speech data is available. Traditionally, Vocal Tract Length Normalization (VTLN) mitigates acoustic mismatch in hybrid systems, while End-to-End (E2E) systems use data augmentation. We investigate speed perturbations, spectral augmentation, and VTLN in E2E CSR systems across Dutch, German, and Mandarin. Our experiments show that speed perturbations and spectral augmentation significantly improve performance, with VTLN offering further enhancements while maintaining adult speech recognition. VTLN benefits both genders and is particularly effective for younger children.
• RQ: How to enhance SCR performance while maintaining performance on adults’ speech when adapting the model to children’s speech?
• Hypothesis: VLTN, speed perturbation, and spectral augmentation can be useful.
• Conclusion: VLTN is used for the 1st times to improve E2E CSR work augmentation and normalization enhance CSR task performance the performance of adult speech is largely preserved similar observations in all 3 languages
• Critical observations: Because VTLN needs to be trained independently and then used as a processing step after feature extraction to warp the features for training the ASR network architecture, it may not be compatible with architectures that utilize raw waveform data rather than features. As a result, integrating VTLN into such architectures requires further exploration.
• Relevance: The study's focus on improving Automatic Speech Recognition (ASR) for children's speech, despite limited annotated data, holds relevance to the endeavor of enhancing ASR performance for older adults. Both populations present challenges due to variability in speech patterns and the scarcity of annotated data. Techniques explored in the study, such as Vocal Tract Length Normalization (VTLN) and data augmentation, offer potential solutions that could be adapted to address age-related changes in older adults' speech. Comparative analyses across languages and considerations of age and gender factors provide valuable insights applicable to developing tailored ASR systems for the older adult population. Overall, the study's methodologies and findings offer valuable parallels and considerations for researchers aiming to improve ASR performance for older adults.

Geng, M., Xie, X., Liu, S., Yu, J., Hu, S., Liu, X., & Meng, H. (2022). Investigation of data augmentation techniques for disordered speech recognition. arXiv preprint arXiv:2201.05562.

• Summary: The final speaker adapted system constructed using the UASpeech corpus and the best augmentation approach based on speed perturbation produced up to 2.92% absolute (9.3% relative) word error rate (WER) reduction over the baseline system without data augmentation, and gave an overall WER of 26.37% on the test set containing 16 dysarthric speakers.
• RQ: systematically investigate different data augmentation techniques for disordered speech recognition.
• Conclusion: It suggests that speed-perturbation based augmentation produces the largest improvement in system performance despite the huge mismatch between normal and disordered speech.
• Critical observations:  They increased the amount of speed perturbation data to four times and six times, with only dysarthric speech being processed, the mean WER showed that four times the amount of the original data made the model performance better than six times (4x: 29.47, 6x: 29.52). More augmented data cannot further improve the model performance. In addition, increasing the augmented data from two to four times only reduced the WER by 0.2%. They did not further increase the amount of augmented data, while according to the results when only dysarthric speech data was augmented, it is doubtful whether more data can still lower the WER. This can be explored in future studies by increasing the amount of augmented data from one to six or more times while keeping all other factors the same.
• Relevance: The study exploring data augmentation techniques for dysarthric speech recognition offers insights applicable to improving ASR performance for older adults. By addressing challenges common to both dysarthric speech and speech from older adults, such as variations in speech patterns and articulation, the study provides valuable methodologies and findings. Specifically, the effectiveness of techniques like speed perturbation-based augmentation in enhancing ASR performance underscores their potential utility in optimizing systems for recognizing older adult speech. Furthermore, the study's identification of augmentation limitations and suggestions for future research pave the way for continued refinement of ASR systems tailored to the unique characteristics of older adult speech.

Synthesis

The articles reviewed collectively contribute to the ASR field, showing a trend towards multimodal data use, context awareness, and noise reduction techniques to address complexities in human speech such as sarcasm and humor. Key observations include the importance of integrating audio, visual, and textual data for better sarcasm detection, the effectiveness of dual-channel noise reduction in vehicular environments, the application of deep learning for respiratory sound classification and speech enhancement in noisy settings, and data augmentation techniques in improving ASR performances for a specific group of speakers. Challenges mentioned across these studies involve data scarcity, handling diverse dialects, and computational demands. Future research directions suggest a focus on improving ASR systems' adaptability across languages, cultures and groups, better managing non-IID and imbalanced data, and enhancing emotional intelligence in speech recognition. These findings indicate ongoing efforts to make ASR technologies more intuitive and effective in complex human-machine interactions.

Contributors

Contributors: A list of contributors by contribution

• Article Wang et al. (2023): Yaling Deng
• Article Sungjoo Ahn and Hanseok Ko (2005): Dongwen Zhu
• Article Zhang and Qian (2023): Dongwen Zhu
• Article Zhou et al. (2024): Dongwen Zhu
• Article Bae et al. (2023): Soogyeong Shin
• Article Gairola et al. (2021): Soogyeong Shin
• Article Yang et al. (2023): Soogyeong Shin
• Article Castro et al. (2019) : Erin Shi
• Article Zhang et al. (2021): Youyang Cai
• Article Patel, T., and Scharenborg, O. (2024): Wansu Zhu
• Article Geng et al. (2022): Wansu Zhu
• Introduction: All
• Synthesis: All

ASR II

Introduction

Briefly introduce your thematic focus and its significance in the field of speech technology.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Wang, C., Wu, Y., Du, Y., Li, J., Liu, S., Lu, L., ... & Zhou, M. (2019). Semantic mask for transformer based end-to-end speech recognition. arXiv preprint arXiv:1912.03010.

• Summary: The article presents a semantic mask-based augmentation approach for improving end-to-end ASR systems. This method involves masking the input features corresponding to specific output tokens, such as words or word-pieces, during training (similar to how BERT is trained with its [MASK] token). The objective is to force the model to predict the masked tokens using contextual information, with this enhancing the model's generalization capabilities. Experiments on the Librispeech 960h and TedLium2 datasets demonstrated state-of-the-art performance, showing the effectiveness of this approach.
• RQ: Can the generalization capacity and language modeling power of end-to-end ASR models be improved with the employment of an NLP technique of semantic masking?
• Hypothesis: By applying a semantic mask to mask out input features corresponding to specific output tokens, the models will be encouraged to rely more on contextual information, improving their modeling capabilities and generalization.
• Conclusion: The introduction of a semantic mask in transformer-based E2E ASR models leads to significant improvements in WER on the Librispeech and TedLium2 datasets. This approach enhances the model's ability to use contextual information and strenghtens its robustness to various acoustic distortions, which potentially can be useful for the task of whispering speech recognition as well.
• Critical observations: The semantic mask approach is particularly effective in challenging conditions, where reliance on contextual information becomes crucial for accurate token prediction, so I may assume it could prove useful in whispering speech too, where one word could be more prominent than the other. However, while the paper describes the semantic masking strategy, further details on how tokens could be selected for masking and the criteria for that could enhance reproducibility and allow for more detaield analysis of why this strategy works.
• Relevance: The paper contributes to the ongoing development of more robust and accurate ASR systems by introducing a novel approach to enhance the language modeling capabilities. Its findings are relevant for researchers in the field of speech recognition, offering a new method for improving the performance of ASR systems, especially in challenging environments.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Wang et al., (2019): Igor Marchenko
• Article XXX: YOUR NAME
• Introduction: All
• Synthesis: All

Speech Enhancement

Introduction

Speech enhancement/restoration represent pivotal areas within the field of speech technology, focusing on the improvement and rehabilitation of speech signals that have been degraded by various factors such as noise, reverberation, and data compression. The significance of this thematic focus cannot be overstated, as it directly impacts the usability, intelligibility, and overall quality of speech communication in diverse contexts, including telecommunications, voice assistants, and hearing aids. This literature collection aims to compile the most recent and influential works that drive innovation in these domains, highlighting the cutting-edge methodologies and the transformative potential they hold for enriching human-computer interaction and ensuring the accessibility of speech-based services for all users.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Donahue, C., Li, B., & Prabhavalkar, R. (2018). Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (arXiv:1711.05747). arXiv. http://arxiv.org/abs/1711.05747

• Summary: This paper investigates the application of Generative Adversarial Networks (GANs) for speech enhancement, particularly for improving the noise robustness of ASR systems. Through comprehensive experiments, it introduces a frequency-domain approach (FSEGAN) to speech enhancement that shows improved ASR performance over traditional time-domain methods (SEGAN).
• RQ: Can GAN-based speech enhancement techniques effectively improve the noise robustness of ASR systems compared to traditional noise suppression methods?
• Hypothesis: The paper hypothesizes that GAN-based speech enhancement, especially when operating on log-Mel filterbank spectra rather than waveforms, will provide significant improvements in ASR system performance in noisy conditions.
• Conclusion: The study concludes that while GAN-based speech enhancement methods, particularly FSEGAN, can improve ASR performance in noisy conditions, they do not outperform multi-style training (MTR) methods. Retraining the ASR system using both the original noisy audio and the audio improved by GANs leads to better performance. This suggests that GAN-enhanced audio could be a valuable addition to improve ASR systems when used alongside the original noisy input.
• Critical observations: SEGAN, while effective in removing additive noise, is less effective in reverberant conditions compared to the frequency-domain approach (FSEGAN). On the contrast, FSEGAN significantly improves ASR performance but does not outperform traditional MTR alone. However, combining noisy and enhanced features for retraining enhances the system's robustness.
• Relevance: This article is relevant to techniques used to bolster the performance of ASR systems, highlighting the significant potential of innovating GAN-based model in this field.

Y. Koizumi, H. Zen, S. Karita, et al. (2023). Miipher: A robust speech restoration model integrating self-supervised speech and text representations, arXiv:2303.01664.

• Summary: The paper presents Miipher, a robust speech restoration (SR) model that integrates self-supervised speech and text representations to enhance the quality of degraded speech signals. The model is designed to address two primary challenges in SR: phoneme masking and deletion.
• RQ: How to develop a robust speech restoration (SR) model that can convert degraded speech signals into high-quality ones, with a focus on handling difficult degradations such as phoneme masking and deletion?
• Hypothesis: The proposed SR model, Miipher, will be robust against various audio degradations and enable the training of high-quality text-to-speech (TTS) models from restored speech samples.
• Conclusion: The study concludes that Miipher is effective in restoring speech samples in-the-wild and can increase the value of speech samples by improving their quality as training data for speech generation tasks.
• Critical observations: The use of w2v-BERT features significantly improves SR performance compared to log-mel spectrogram-based methods, the effectiveness of PnG-BERT features in preserving text content, and the importance of speaker embeddings for retaining speaker characteristics in restored speech.
• Relevance: The relevance of this study is significant for the field of speech enhancement/restoration, as it demonstrates a method to enhance the quality of existing speech datasets and expand the potential applications of non-studio speech recordings.

Contributors

• Introduction: Janice Huang

• Article Donahue et al.(2018): Ting Zhang
• Article Nitya Tiwari (2020): Ziyun Zhang
• Article Y. Koizumi et al.(2023): Janice Huang
• Synthesis:

Miscellaneous

This last section corresponds to articles that did not fit well inside other themes.

Introduction

Voice technology, transcending the traditional boundaries of speech recognition and synthesis, has emerged as a transformative force in a multitude of sectors, revolutionizing not just how we communicate with machines, but also how sound is manipulated and perceived in our digital world. This segment, aptly titled "None of the Above," delves into the innovative applications of voice technology beyond the realms of text-to-speech (TTS) and automatic speech recognition (ASR). It encompasses a wide array of technologies including voice enhancement, noise reduction, accent modification, and speaker seperation, each playing a pivotal role in refining and enriching the auditory experience. These advancements underscore the versatility and depth of voice technology, pushing the boundaries of what is possible in audio quality, clarity, and customization.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Speech Emotion Recognition

Grimm, M., Kroschel, K., & Narayanan, S. (2007, April). Support vector regression for automatic recognition of spontaneous emotions in speech. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07 (Vol. 4, pp. IV-1085). IEEE.

• Summary: The paper presents methods for estimating emotions expressed spontaneously in speech, using Support Vector Regression (SVR). It evaluates three emotion primitives—valence, activation, and dominance—showing SVR's superiority over Fuzzy Logic and Fuzzy k-Nearest Neighbor classifiers in accuracy and correlation with human assessments.
• RQ: How to estimate emotions under the conditions of (1) nonacted, spontaneous speech and (2) non-categorical, quasicontinuous emotional content.
• Hypothesis: SVR can more accurately estimate emotions in speech compared to traditional classifiers, given its ability to handle continuous emotion primitives and complex non-linear relationships in data.
• Conclusion: SVR outperforms Fuzzy Logic and k-Nearest Neighbor classifiers in estimating emotions from speech, achieving lower classification errors and higher correlations with reference emotions. This underscores SVR's suitability for continuous-valued emotion estimation in spontaneous speech.
• Critical observations: SVR yields the lowest mean classification errors and highest correlation coefficients for emotion estimation. In addition, Feature selection indicates that using 20 features suffices for accurate emotion estimation across different classifiers.
• Relevance: This study advances automatic emotion recognition in speech, crucial for improving human-machine interaction and developing emotionally intelligent systems. Future work will investigate designing a real-time system using the algorithms. The advantage of continuous-valued estimates of the emotional state of a person could be used to build an adaptive emotion tracking system that is capable to adapt to individual personalities and long-term moods.

Synthetically improving foreign-accented speech recognition

Introduction

More often than not, speech corpora either contain only native speech, or the non-native subset is significantly underrepresented. At the same time, gender and foreign accent are the most salient factors contributing to changes in the acoustics of speech. However, not only are there numerous possible combinations of L1 and L2s, but the annotation and labelling os recordings to a suitable degree (e.g. age of L2 acquisition, country of origin, L1, L2 proficiency, language of education etc. are all factors that should be reported in order to make the speech resources reliable and usable) are laborious and expensive.

In light of these challenges, methods of synthetical data augmentation have been recently explored in the literature. While creating synthetically-accented data through accent conversion models (ACMs) is a straightforward, inexpensive, and oof-the-shelf approach, it is not without limitations and the degree to which recognition performance is improved through such approaches depends on several factors. The following three articles provide some insight into these approaches and highlight both major advantages and persistent challenges.

Zhao et al. (2018): Accent conversion using phonetic posteriograms

Summary: Accent conversion (AC) means transforming non-native speech to sound as if the speaker had a native accent, or vice-versa. The main challenge faced in traditional methods of voice conversion is decoupling the speaker’s voice quality from their pronunciation (i.e. teasing apart accent information and keeping everything else acoustically unchanged). Additionally, when mapping source spectra from a native speaker into the acoustic space of an L2 speaker, previous attempts focus on acoustic similarity: changing formants- and pitch trajectories, blending spectral envelopes. The alternative used here is, in turn, is phonetic similarity, which maps source to target based on an intermediate phonetic label. The phonetic posteriograms are computed using a DNN-based acoustic model. The distance between these phonetic posterior feature vectors is calculated to find the closest pairs of frames between source (native) and target (L2) speakers. The frame pairs are used to train a GMM. The two baselines used are acoustic similarity matching and dynamic time warping.

Experimental setup: get Kaldi DNN acoustic model, train it on Librispeech data, get native English speech (CMUArctic) and non-native recordings (Hindi, Korean, Arabic), use STRAIGHT for speech decomposition, MFCC extraction, train GMMs (128 components), synthesize speech by reconstructing spectrograms and adding aperiodicity.

RQ: How can accent-related features be successfully decoupled from speaker-related features, to achieve non-native to native voice conversion while preserving speech quality?

Results: Synthesized results were compared to baselines through listening tasks using Mechanical Turk (rating acoustic quality, speaker identity y/n, nativeness of resynthesized speech):

• significantly higher acoustic quality ratings compared to baselines.
• comparable speaker identity scores.
• strong preferrence for posteriogram conversions by native EN speakers as more ‘native-like‘ compared to baselines and original L2 utterances.

Critical observations: This paper addressed the opposite issue, namely converting foreign-accented speech to sound like native one (mainly for educational purposes). This still means you need to figure out which features are related to accent, and which features are related to anything else, but is arguably the easier thing to do, as it requires to drop information instead of successfully adding it. Additionally, the approach is not entirely explainable, because posteriograms are encoder features and it's not always transparent what is learned to be most relevant. Lastly, this approach likely works increasingly worse the fewer speakers there are in a dataset. Even if you accented speech data, one speaker can only have one accent, so in case the number of speakers is small, the model might learn to encode speaker identity instead of accent features.

Relevance: It is important to know that given enough speakers and enough data, accent features can be decoupled from other speech features and dropeed to obtain a higher perceived 'nativeness' of the speech.

Jin et al. (2023): Voice-preserving zero-shot multiple accent conversion

Summary: Separating accent from speaker identity is usually the hardest, because each speaker in the dataset has one single accent. Previous attempts at doing this include:

• use adversarial learning to get a discriminator to wipe out speaker-dependent information from content embeddings.
• quantization of different features in speech to obscure undesired information.

The main problem with conventional approaches to conversion is that they very often require available utterances with the same text in both source and target accent, making their applicability very limited. Alternatively, different approaches require either training or fine-tuning on the input utterances.

The current paper uses a pronunciation encoder, an acoustic encoder, and a HiFiGAN voice decoder. During training, the model minimises reconstruction loss between input and output mel-spectrograms. The pronunciation encoder synthesizes accent-dependent pronunciation sequences using accent IDs. The acoustic encoder mapss MFCCs and periodicity features to a single vector, while adversarial training removes accent information. Lastly, the decoder reconstructs waveforms from the processed features. The model is evaluated on audio quality, speaker similarity, and accent conversion effectiveness.

Results: Results indicate it maintains comparable audio quality to the original, maintains speaker similarity, and is efficient in replicating perceived nativeneess. However, listeners struggled to identify synthesized accents if they were unfamiliar with the target language (e.g. a native US listener could not classify a Korean accent on English as such, but a bilingual Korean-American listener could). Overall, the paper presents one of the best performing ACMs, that is able to preserve both speaker identity and acoustic quality during conversion.

Critical observations: I think this paper achives a lot given that it's zero shot, but I am a bit critical about just how 'zero-shot' it truly is. They use a pre-trained acoustic model and while they do not require accent labels or speaker IDs, it seems that their training set contains over 24h of accented speech for all accents that they're synthesizing in. Additionally, none of their code is openly available, which is understandable for a private corporation like Meta, but it's still a bit disappointing.

Klumpp et al. (2023): Synthetic cross-accent data augmentation for ASR

Summary: Foreign-accentes speech is usually underrepresented in, if not absent from speech corpora. Auxiliary input (learned accent embeddings, intermediate wav2vec2.0 representations) can address the decreased ASR recognition on this type of speech; the challenge remains that of achieving good accent conversion while preserving source speaker voice characteristics. The current approach builds on a pre-existing ACM by Jin et al. (2023) -- see above -- and aims to provide synthetic ASR training data using it. Phonetic knowledge is crucially injected into training to improve accent-specific pronunciation, and learnable accent representations are introduced to allow for variable accent strengths and adaptability to unseen accents.

The experimental setup involved evaluating two ASR models using Librispeech data. The first model (Base) utilized an efficient memory transformer followed by a recurrent neural transducer (RNNT), while the second model (HuBERT) had a similar structure with adjustments in channel configurations and dropout probabilities. The ASR models were tested on Librispeech data and accents from L2-Arctic corpus and Accented Vox Populi (AVP) dataset.

In experiments, the baseline ASR systems were trained without synthetic accented speech data, then evaluated. Three additional ASR models were trained with a combination of real and synthetic accented data, using a ratio of 80% real and 20% synthetic data. The ratio remained consistent across all accents. Finally, learned accent embeddings from L2-Arctic samples were visualized using t-SNE plots to assess their suitability for encoding accent information in an Accent Conversion Model (ACM).

RQ: Is it possible to improve ASR of accented speech with synthetic samples of a particular accent?

Results: The inclusion of one synthetic accent during ASR training had a positive effect on recognition results for that particular accent, which was a clear indicator that the ACM was able to synthesize a sufficient degree of accentedness. At the same time, HuBERT'd performance decreased with the use of synthetic data, likely due to the fact that it was not pre-trained on any and fine-tuning did not do enough. The Base model, which was trained from scratch, had a much grater benefit from the synthetic data. Notably, even when all seven accents were introduced in training, this did not improve performance on other unseen accents.

Overall, including one synthetic accent improved performance on that accent; and including several accents improved performance on those accents, but none of the conditions improved recognition on accents not seen in training. Additionally, pre-trained HuBERT did not benefit much from additional synthetic data fine-tuning, whereas a model trained from scratch saw much greater benefit from this approach.

Critical observations: Again, none of this replicable because the code is not available. It would have been also interesting to see a bit more ASR models be tested on this; this particular comparison does highlight the pre-trained/trained from scratch distinction in performance on this task, but there are other models that are seemingly good candidates and were not included.

Relevance: The authors show the potential for using synthetically accented data as a data augmentation approach to improve ASR performance on foreign-accented speech.

General insights

The synthesis of accented speech as a data augmentation method in ASR is promising for improving recognition performance on non-native speech. The three articles reviewed provide valuable insights into accent conversion methods and their implications for ASR systems. Zhao et al. (2018) shows the effectiveness of phonetic posteriograms in converting foreign-accented speech to sound more native-like and successfully decouples accent-related features from other speech characteristics. Jin et al. (2023) proposed a zero-shot multiple accent conversion approach, maintaining audio quality and speaker identity during conversion, albeit with limitations in accent classification for unfamiliar listeners. Klumpp et al. (2023) extended this work by integrating synthetic accented speech data into ASR training, showing improvements in recognition performance on the trained accents. However, the effectiveness varied depending on the model architecture, with pre-trained models benefiting less from synthetic data than models trained from scratch. Despite promising results, the lack of code availability and limited generalizability to unseen accents pose challenges for broader adoption. Overall, while accent conversion models offer a promising strategy for data augmentation in ASR, further research should focus on generalization and replicability for real-world applications.

References

Jin, M., Serai, P., Wu, J., Tjandra, A., Manohar, V., & He, Q. (2023, June). Voice-preserving zero-shot multiple accent conversion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

Klumpp, P., Chitkara, P., Sarı, L., Serai, P., Wu, J., Veliche, I. E., ... & He, Q. (2023). Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition. arXiv preprint arXiv:2303.00802.

Zhao, G., Sonsaat, S., Levis, J., Chukharev-Hudilainen, E., & Gutierrez-Osuna, R. (2018, April). Accent conversion using phonetic posteriorgrams. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5314-5318). IEEE.

Accent Modification

Introduction

Accents play a crucial role in shaping the unique characteristics of speech, reflecting an individual's linguistic background and cultural identity. However, the presence of foreign accents can sometimes pose challenges, particularly in the speaking test for language proficiency assessment.

Finkelstein, L., Zen, H., Casagrande, N., Chan, C., Jia, Y., Kenter, T., Petelin, A., Shen, J., Wan, V., Zhang, Y., Wu, Y., & Clark, R. (2022). Training Text-To-Speech Systems From Synthetic Data: A Practical Approach For Accent Transfer Tasks. Google LLC. Retrieved from https://arxiv.org/abs/2208.13183

Summary: This paper presents a practical approach for accent transfer tasks in text-to-speech (TTS) synthesis, where aspects of one speaker's speech are transferred to another speaker's speech. The authors address the challenge of creating high-quality transfer models that are also stable and suitable for user-facing applications. They propose a two-step training process involving a Tacotron-based accent transfer model and a robust CHiVE-BERT TTS system. The CHiVE-BERT system is trained on synthetic data generated by the Tacotron model, which results in high-quality audio with transferred accents while preserving speaker characteristics.

RQ: How can text-to-speech systems be trained to achieve accent transfer effectively and stably, without compromising the quality or usability of the synthesized speech?

Hypothesis: By training a robust TTS system on synthetic data generated by a less stable but high-quality accent transfer model, it is possible to achieve a balance between quality and stability in accent transfer tasks.

Conclusion: The study concludes that the proposed two-step training approach, using synthetic data generated by a Tacotron-based model to train a CHiVE-BERT system, yields reliable performance in terms of naturalness and accent transfer capability. The quality loss associated with the switch to synthetic data is within acceptable bounds, and the final system produces high-quality audio that maintains the original speakers' characteristics.

Critical observations: The authors note that the quality of the final system is affected by the intermediate Tacotron model, with some accents showing significant quality loss, particularly for female speakers in British English. Training on synthetic data can result in lower quality loss compared to using human recordings, possibly due to the reduced variance in synthetic data. The choice of vocoder, synthesizer, and the balance between synthetic and human recordings are critical in the training process, with the final system benefiting from a combination of both.

Relevance: The research on accent transfer in TTS systems aligns closely with my focus on accent modification for Turkish immigrants in Dutch oral exams. The methodologies explored for synthesizing and transferring accents can be adapted to develop tools that neutralize accents, enhancing exam fairness by ensuring evaluations are based on language skills rather than accent.

Li, W., Tang, B., Yin, X., Zhao, Y., Li, W., Wang, K., Huang, H., Wang, Y., & Ma, Z. (2020). Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech. arXiv preprint arXiv:2005.09271. Retrieved from https://arxiv.org/abs/2005.09271

Summary: This paper presents an end-to-end accent conversion framework aimed at transforming non-native accents into native accents while preserving the speaker's voice timbre. The proposed system introduces reference encoders to utilize multi-source information and optimizes the model architecture using GMM-based attention for improved synthesized performance. Experimental results show significant improvements in acoustic quality and native accent while retaining the non-native speaker's voice identity.

RQ: How can accent conversion be improved to better transform non-native accents into native accents in a way that maintains the original speaker's voice identity?

Hypothesis: Incorporating reference encoders and optimizing the model architecture with GMM-based attention will enhance the quality and naturalness of converted speech, leading to more effective accent conversion.

Conclusion: Incorporating reference encoders and optimizing the model architecture with GMM-based attention will enhance the quality and naturalness of converted speech, leading to more effective accent conversion.

Critical observations: The paper highlights the importance of prosodic and expressive information in accent conversion, which is effectively captured by the reference encoder. The GMM-based attention mechanism is found to be more stable and powerful for feature representation compared to traditional windowed attention.

Relevance: The research is relevant to accent modification efforts, particularly in language learning and pronunciation training contexts. The proposed accent conversion techniques could be applied to develop tools that help non-native speakers improve their pronunciation and reduce their accents, thereby enhancing communication and integration in societies where the target language is spoken natively.

Zang, X., Weng, F., & Zang, X. (2022). Foreign Accent Conversion using Concentrated Attention. In 2022 IEEE International Conference on Knowledge Graph (ICKG). Retrieved from https://ieeexplore.ieee.org/document/978-1-6654-5101-7

Summary: This paper introduces a novel method for foreign accent conversion (FAC) utilizing Phonetic Posteriorgrams (PPGs) and Log-scale Fundamental frequency (Log-FO) to address phonetic and prosody mismatches. The proposed approach employs concentrated attention to enhance the alignment of input sequences and mel-spectrograms, selecting the top k highest score values in the attention matrix row by row. The method is evaluated through objective metrics and demonstrates improved voice naturalness, speaker similarity, and accent similarity.

RQ: How can foreign accent conversion be improved to achieve better alignment and naturalness in synthesized speech while preserving the source speaker's identity?

Hypothesis: Implementing concentrated attention in the foreign accent conversion process will result in more accurate alignment of input sequences with mel-spectrograms, leading to improved accent conversion quality and naturalness in synthesized speech.

Conclusion: The proposed method using concentrated attention for foreign accent conversion delivers comparable or better results than previous methods in terms of voice naturalness and accent similarity. The concentrated attention mechanism effectively focuses on the most relevant frames for better alignment and synthesized speech quality.

Critical observations: The concentrated attention mechanism is found to be beneficial for achieving better alignment between input sequences and target sequences, resulting in improved speech synthesis.

Relevance: The research is relevant to the field of speech synthesis and voice conversion, particularly for applications that require the alteration of accents while maintaining the original speaker's voice characteristics. This work contributes to the development of systems that can aid in language learning, dubbing, and other scenarios where accent modification is beneficial, enhancing the quality and naturalness of synthesized speech.

Speech Separation

Zegers, J. (2019). CNN-LSTM models for multi-speaker source separation using Bayesian hyperparameter optimization. arXiv preprint arXiv:1912.09254.

Summary: This paper explores the use of Bayesian hyperparameter optimization for parallel CNN-LSTM models in the task of multi-speaker source separation (MSSS). Experiments were conducted with mixtures from the WSJ0 corpus and found that parallel CNN-LSTM models performed better than individual CNN or LSTM models.

Research Question (RQ): How does Bayesian hyperparameter optimization affect the performance of parallel CNN-LSTM models in multi-speaker source separation?

Hypothesis: The hypothesis was that the Bayesian optimization technique would find a better hyperparameter set that allows the parallel CNN-LSTM model to outperform individual CNNs or LSTMs in MSSS.

Conclusion: The study concluded that models with more trainable parameters in the LSTM portion performed better and that parallel CNN-LSTM models with Bayesian hyperparameter optimization outperformed the other models tested.

Critical Observations: The LSTM part of the model was crucial for performance, and bidirectional LSTMs performed better than unidirectional ones. Also, the study noted that more trainable parameters in the LSTM were generally preferred.

Relevance: This research is relevant for advancements in speech processing, specifically in improving source separation techniques which is a foundational task in many audio processing applications.

Isik, Y., Roux, J. L., Chen, Z., Watanabe, S., & Hershey, J. R. (2016). Single-channel multi-speaker separation using deep clustering. arXiv preprint arXiv:1607.02173

Summary: This study improved the baseline system for speaker-independent multi-speaker separation using deep clustering with an end-to-end signal approximation objective. By optimizing the model with enhancements like regularization, larger temporal context, and a deeper architecture, significant improvements in signal-to-distortion ratio and word error rate were achieved.

Research Question (RQ): Can the performance of speaker-independent multi-speaker separation be improved by using deep clustering with an end-to-end training approach?

Hypothesis: The authors hypothesized that incorporating an end-to-end signal approximation objective would lead to better performance in speech separation.

Conclusion: The paper concluded that the deep clustering approach with an end-to-end signal approximation objective greatly improved signal quality metrics and reduced speech recognition error rates, contributing to solving the cocktail party problem.

Critical Observations: The model performed well even with different numbers of speakers, and the addition of a signal approximation objective substantially reduced the word error rate when integrated with automatic speech recognition systems.

Relevance: This research contributes to solving complex audio environments' speech recognition challenges, aiding the development of better voice-activated systems that can function effectively in real-world conditions.

Maiti, S., Ueda, Y., Watanabe, S., Zhang, C., Yu, M., Zhang, S., & Xu, Y. (2023). EEND-SS: Joint end-to-end neural speaker diarization and speech separation for flexible number of speakers. In 2022 IEEE Spoken Language Technology Workshop (SLT) (pp. 480-487). IEEE.

Summary: The paper presents EEND-SS, a framework that integrates speaker diarization, speech separation, and speaker counting into a single end-to-end trainable model. It demonstrated improved performance over single-task models and enhanced speaker counting for a flexible number of speakers.

Research Question (RQ): Can an integrated framework that combines speaker diarization and speech separation improve performance over models that address these tasks separately?

Hypothesis: The authors posited that a joint model incorporating speaker diarization, speech separation, and speaker counting would perform better than individual models tackling each task separately.

Conclusion: The study concluded that the EEND-SS framework could outperform single-task baselines in both diarization and separation metrics and improved speaker counting performance.

Critical Observations: A key observation was that jointly learning to separate and diarize helped the model perform better in diarization, particularly in less overlapped conditions, suggesting better generalization.

Relevance: The results of this study are highly relevant for multi-speaker environments, improving the performance and applicability of voice recognition systems in scenarios with a variable number of speakers. Each of these studies contributes to the field of speech processing, advancing our understanding and capability in separating and recognizing speech in challenging audio scenarios.

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Finkelstein et al.(2022): Chenyu Li
• Article Li et al.(2020): Chenyu Li
• Article Zang et al(2022): Chenyu Li
• Article Grimm et al.(2007): Yining Lei
• Introduction: Chenyu Li
• Synthesis: All

Subsections:

The section Synthetically improving foreign-accented speech recognition was written by Maria Tepei.

The section Accent Modification was written by Chenyu Li

The section Speech Separation was written by Sherry Yu-Ting Yeh