State-of-the-art

Theme: Template copy/paste but do not delete

Introduction

Briefly introduce your thematic focus and its significance in the field of speech technology.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Jones et al. 2023: YOUR NAME
• Article XXX: YOUR NAME
• Introduction: All
• Synthesis: All

Low-resource ASR

Introduction

Our theme focuses on automatic speech recognition (ASR) of low-resource languages. Low-resource languages are often underrepresented in ASR due to the limited amount of data, limited amount of speakers, and low commercial impact. However, it is important for both preserving and encouraging the use of low-resource languages to allow for users to utilize ASR for their own language. Therefore, our theme is significant in the field of speech technology. A similar data scarcity challenge exists for ASR of dysarthric speech, which occurs in neurodegenerative disorders like Parkinson's disease. Some transfer learning techniques has been explored to improve speech systems for dysarthric and low-resource languages respectively by leveraging data from other languages/domains. Such cross-domain transfer learning shows promise, but requires careful study to effectively bridge the data gaps.

Article summaries

Wang, S., Rohdin, J., Plchot, O., Burget, L., Yu, K., & Cernocky, J. (2020). Investigation of Specaugment for Deep Speaker Embedding Learning. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7139–7143. https://doi.org/10.1109/ICASSP40776.2020.905348

• Summary: The article investigates the effectiveness of SpecAugment, a data augmentation method, for speaker verification tasks using TDNN and ResNet34 models with Softmax and AAMSoftmax loss functions. Experiments on NIST SRE 2016 Cantonese and Tagalog subsets and Voxceleb1 dataset show improved performance with SpecAugment, achieving 3.72% and 11.49% EER for NIST SRE 2016 Cantonese and Tagalog, respectively, and 1.47% EER for Voxceleb1. SpecAugment demonstrates promising results for speaker verification across different languages, enhancing system robustness without complex offline augmentation.
• RQ: How effective is SpecAugment, a data augmentation method originally proposed for speech recognition, when applied to speaker verification tasks across different languages, specifically Cantonese and Tagalog?
• Hypothesis: Applying SpecAugment, a data augmentation technique initially developed for speech recognition, to speaker verification tasks will lead to performance improvements across different languages, including Cantonese and Tagalog.
• Conclusion: Implementing SpecAugment for speaker verification tasks yields significant performance improvements across different languages. Specifically, the study demonstrates that SpecAugment, applied on-the-fly without complex offline augmentation methods, achieves state-of-the-art results in speaker verification tasks for Cantonese and Tagalog, as well as for the Voxceleb1 dataset.
• Critical observations: The critical observation of the article focuses on the implementation of SpecAugment for speaker verification tasks across various languages, particularly Cantonese and Tagalog, which are considered low-resource languages. The study demonstrates that SpecAugment, applied on-the-fly, effectively improves performance in speaker verification tasks for these languages, achieving significant reductions in Equal Error Rate (EER) compared to traditional methods. This highlights the potential of SpecAugment as a simple yet powerful augmentation technique, particularly beneficial for low-resource language processing tasks.
• Relevance: The relevance of the article to the topic of low-resource language Automatic Speech Recognition (ASR) lies in its exploration of SpecAugment as a data augmentation technique for speaker verification tasks in languages like Cantonese and Tagalog, which are considered low-resource. By demonstrating the effectiveness of SpecAugment in improving performance in speaker verification tasks for these languages, the study showcases a potential solution to the challenges posed by limited data availability in low-resource language ASR. This highlights SpecAugment as a valuable tool for enhancing ASR systems' robustness and accuracy in under-resourced linguistic contexts.

Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., ... & Wu, Y. (2023). Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037.

• Summary: Google's Universal Speech Model aims to develop an ASR model that will be able to perform speech recognition on all languages of the world. This paper leverages large amounts of unlabelled speech and text data from YouTube to train a multilingual-encoder that can then be used in fine-tuning on very small amounts of labelled data. This allows them to outperform Whisper with significantly less labelled data, while also showing that this approach works positively for lower-resource languages.
• RQ: Can we leverage the large amounts of unlabelled speech data to perform massively multilingual ASR and speech translation?
• Hypothesis: By using a vast amount of unlabelled data, the encoder will learn speech representations that can be leveraged in fine-tuning and downstream tasks.
• Conclusion: Pre-training on unlabelled data is an effective way to improve multilingual performance while requiring much less labelled data.
• Critical observations: Although they keep mentioning that their performance is stellar on low-resource languages, no results were presented for these languages specifically. Most results are from multilingual datasets that might be imbalanced as well. Furthermore, the models and training data are not publicly available, making the research less approachable for improvements.
• Relevance: This paper is highly relevant for our theme as it aims to improve low-resource ASR through unlabelled data, which is an effective solution to the data scarcity problem.

Zhang, Y., Herygers, A., Patel, T., Yue, Z., & Scharenborg, O. (2023). Exploring data augmentation in bias mitigation against non-native-accented speech (arXiv:2312.15499). arXiv. http://arxiv.org/abs/2312.15499

• Summary: The study aimed to investigate the impact of data augmentation techniques on the performance of Flemish Automatic Speech Recognition (ASR) systems for both native Flemish speakers and those with non-native accents. Specifically, the research focused on addressing biases against non-native-accented Flemish speech. Various data augmentation methods were applied to augment the training data, and the performance of the ASR system was evaluated using both native and non-native speakers' speech samples. The results suggested that tailored data augmentation techniques can lead to improved ASR system performance for both native and non-native-accented Flemish speech. This finding highlights the potential of data augmentation in mitigating bias and enhancing the accuracy of ASR systems across diverse speaker demographics.
• RQ: What is the optimal type of data augmentation, in terms of reducing bias against non-native-accented Flemish in a Flemish ASR system, when applied to both native Flemish and non-native-accented Flemish?
• Hypothesis: Applying specific types of data augmentation techniques, tailored to address bias against non-native-accented Flemish speech, will lead to improved performance in a Flemish Automatic Speech Recognition (ASR) system for both native Flemish and non-native-accented Flemish speakers.
• Conclusion: The study concluded that employing tailored data augmentation techniques can significantly improve the performance of Flemish Automatic Speech Recognition (ASR) systems, particularly in mitigating biases against non-native-accented speech. By augmenting the training data with techniques specifically designed to address the characteristics of non-native accents, the ASR system demonstrated notable enhancements in accuracy for both native and non-native speakers. These findings underscore the importance of considering diversity in training data and utilizing appropriate augmentation strategies to enhance the robustness and inclusivity of ASR systems.
• Critical observations: The performance of Flemish Automatic Speech Recognition (ASR) systems can be significantly improved through the use of tailored data augmentation techniques. Specifically, augmenting the training data with methods designed to address the characteristics of non-native accents resulted in notable enhancements in accuracy for both native and non-native speakers. This observation highlights the importance of considering diversity in training data and employing appropriate augmentation strategies to enhance the inclusivity and robustness of ASR systems.
• Relevance: Low-resource languages often suffer from limited available data for training ASR systems, which can lead to poor performance, especially for speakers with non-native accents. This study demonstrates that tailored data augmentation techniques can substantially improve the accuracy of ASR systems, even in scenarios with limited training data.By addressing the challenges faced by speakers with non-native accents, the paper contributes valuable insights into how ASR technology can be adapted and optimized for low-resource languages. It underscores the importance of developing strategies that account for linguistic diversity and accent variations, ultimately making ASR systems more inclusive and effective in diverse linguistic contexts. Therefore, the findings of this study are highly relevant for researchers and practitioners working on ASR for low-resource languages, offering practical approaches to enhance system performance and usability in such settings.

Wang, H., Wang, S., Zhang, W. Q., & Bai, J. (2023). Distilxlsr: A light weight cross-lingual speech representation model. arXiv preprint arXiv:2306.01303.

• Summary: The authors introduce a compression scheme for multilingual self-supervised speech representation models aimed at enhancing speech recognition performance for low-resource languages while reducing model size for industrial applications. Experiments across two types of teacher models and 15 low-resource languages demonstrate that this method can reduce parameters by 50% while maintaining cross-lingual representation capabilities. The approach is shown to be generalizable across various languages and teacher models, with potential to improve the cross-lingual performance of English pretrained models. Key observations include the effectiveness of data splicing, the importance of layer-jumping initialization, the balance between model compression and performance, and underfitting challenges in low-resource scenarios.
• RQ: The paper investigates how to compress multilingual self-supervised speech representation models, specifically aiming to enhance speech recognition performance for low-resource languages while reducing the model size for easier industrial application.
• Hypothesis: It's possible to significantly reduce the size of multilingual speech representation models without substantially sacrificing performance across various languages by distilling cross-lingual models using only English data and applying techniques such as random phoneme shuffling, layer-jumping initialization, and data splicing.
• Conclusion: The proposed DistilXLSR model successfully reduces parameter size by 50% while maintaining cross-lingual representation capabilities across 15 low-resource languages. This model demonstrates its effectiveness through experimental results, showing comparable performance to larger teacher models and the potential for generalizability and improvement in cross-lingual performance of English pre-trained models.
• Critical Observations:
  1. Randomly shuffling syllables within utterances to reduce linguistic information proved effective for distilling models with cross-lingual capabilities using only English data.
  2. This novel method of initializing student models by leveraging teacher models' pre-trained weights across layers enhances the learning and representation ability of the distilled model.
  3. The study highlights a trade-off between model size and performance, where the distilled models show only slight degradation in performance despite a significant reduction in size.
  4. The paper acknowledges challenges like underfitting, especially evident in datasets with lower quality audio, suggesting that further research could explore structured pruning or other methods to mitigate this.
• Relevance: By employing a novel distillation approach that leverages English data, this model addresses the challenge of accessing and formatting training data across multiple languages, which is particularly difficult for low-resource languages. The effectiveness of DistilXLSR in maintaining performance across 15 low-resource languages, despite a substantial reduction in model size, showcases its potential in breaking down language barriers and enabling more equitable access to speech technology worldwide.

Gandhi, S., von Platen, P., & Rush, A. M. (2023). Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling. arXiv preprint arXiv:2311.00430.

• Summary: The study introduces a novel approach to compressing pre-trained large speech recognition models for efficient deployment in low-resource environments. By leveraging large-scale pseudo-labeling, the research achieves a smaller variant, Distil-Whisper, which significantly reduces the model size and inference time without considerably sacrificing performance. This method particularly benefits low-resource languages by maintaining robustness across various acoustic scenarios and demonstrating potential in extending sophisticated ASR capabilities to languages with limited training data.
• RQ: How can the size of pre-trained speech recognition models, specifically the Whisper model, be reduced for efficient deployment in low-latency or resource-constrained environments while maintaining model robustness and performance?
• Hypothesis: By using pseudo-labelling to create a large-scale open-source dataset and applying a simple word error rate (WER) heuristic to select only the highest quality pseudo-labels for training, it is possible to distill the Whisper model into a smaller variant (Distil-Whisper) that is significantly faster and more parameter-efficient without substantially sacrificing performance.
• Conclusion: Distil-Whisper successfully demonstrates the feasibility of distilling a large-scale speech recognition model into a significantly smaller and faster version without substantial loss in performance. The distilled model achieves a WER performance within 1% of the original Whisper model on out-of-distribution test data, maintains robustness against difficult acoustic conditions, and reduces the propensity for hallucination errors in long-form audio. Furthermore, Distil-Whisper, when paired with Whisper for speculative decoding, offers a significant speed-up in inference times while ensuring identical outputs to the original model.
• Critical Observations: The approach underscores the effectiveness of large-scale pseudo-labelling and a straightforward WER-based heuristic in filtering training data for distillation purposes. The research highlights a crucial balance between model size, speed, and performance robustness, contributing to practical speech recognition applications, especially in constrained environments.
• Relevance: The methodology demonstrates potential for extending sophisticated ASR capabilities to languages with fewer resources by leveraging transfer learning and pseudo-labeling techniques.

Yang, M., Tjandra, A., Liu, C., Zhang, D., Le, D., & Kalinli, O. (2023, June). Learning asr pathways: A sparse multilingual asr model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

• Summary: This research proposes a sparse multilingual ASR model, ASR pathways, which employs language-specific sub-networks to effectively manage multilingual speech recognition without significant performance drops in low-resource languages. The model utilizes iterative magnitude pruning (IMP) and the Lottery Ticket Hypothesis (LTH) to learn language-specific masks, facilitating knowledge transfer and improved performance in languages with scant data. This method enhances the accessibility of advanced ASR technologies in multilingual contexts, showing promise in scaling speech recognition capabilities across diverse language landscapes, including those with fewer resources.
• RQ: How can neural network pruning be optimized for multilingual Automatic Speech Recognition (ASR) without significantly degrading recognition performance on certain languages, given that language-agnostic pruning may discard important language-specific parameters?
• Hypothesis: It's possible to construct a sparse multilingual ASR model, referred to as ASR pathways, which activates language-specific sub-networks or "pathways" for different languages. This approach enables both language-specific optimization and the shared learning of parameters across languages, particularly benefiting lower-resource languages through joint multilingual training.
• Conclusion: The ASR pathways model, which utilizes sparse sub-networks tailored for specific languages within a unified parameter set, outperforms both dense models and language-agnostically pruned models. It demonstrates improved performance on low-resource languages compared to monolingual sparse models, showcasing the effectiveness of this sparse multilingual ASR framework in achieving efficient and robust speech recognition across multiple languages.
• Critical observations: The study found that language-specific pruning masks, developed through Iterative Magnitude Pruning (IMP) or Lottery Ticket Hypothesis (LTH), are crucial for the model's success. These masks enable the model to maintain or even improve performance across languages by preserving essential language-specific parameters while also benefiting from shared knowledge. The LTH approach, in particular, showed superior performance, even with fewer total effective parameters, highlighting the importance of the initial parameter selection in the pruning process.
• Relevance: The shared parameters between these language-specific pathways facilitate knowledge transfer during joint multilingual training, which is especially beneficial for languages with limited training data. The empirical results showing improved performance on low-resource languages compared to monolingual sparse models underline the potential of this method to bring high-quality ASR technologies to low-resource settings.

N, K. D., Wang, P., & Bozza, B. (2021). Using Large Self-Supervised Models for Low-Resource Speech Recognition. Interspeech 2021, 2436–2440. https://doi.org/10.21437/Interspeech.2021-631

• Summary: This paper investigates the effectiveness of using large self-supervised pre-trained models (such as wav2vec 2.0) for low-resource speech recognition tasks. The authors conducted experiments on three Indian languages (Telugu, Tamil, and Gujarati), using different pre-trained models (monolingual English, multilingual) and compared different fine-tuning strategies (CTC, seq2seq, etc.).
• RQ: For low-resource speech recognition tasks, how effective are large self-supervised pre-trained models (such as wav2vec 2.0) compared to traditional supervised learning methods? For Indian languages, are cross-lingual multilingual pre-trained models or monolingual English pre-trained models more suitable? How do different fine-tuning strategies (CTC vs seq2seq) affect model performance? Additionally, how well do these pre-trained models generalize to seen and unseen languages?
• Hypothesis:
  1. Large self-supervised pre-trained models will outperform supervised learning models under low-resource conditions.
  2. Cross-lingual multilingual pre-trained models will perform better than monolingual English models on these Indian languages.
  3. Adopting the CTC fine-tuning strategy will achieve better performance than the seq2seq strategy.
• Conclusion:The multilingual pre-trained model XLSR outperformed the monolingual models on all three languages; for seen languages (like Tamil), the pre-trained model can approach the best performance with only 50% of the training data; the CTC fine-tuning framework performed better than the seq2seq framework, possibly due to the small amount of data; even smaller English pre-trained models showed decent transfer performance on Indian languages.
• Critical observations:The authors did not explain why the larger English pre-trained model underperformed compared to the smaller one, and analysis of the multilingual fine-tuning strategy was limited, only compared to the monolingual strategy. In addition, the impact of different pre-training corpora on model performance was not explored.
• Relevance:This work is important for low-resource speech recognition domains in developing countries. Leveraging large self-supervised pre-trained models can make full use of unlabeled data, alleviating the bottleneck of limited labeled data. This study provides an effective solution for low-resource speech recognition tasks.

Yi, C., Wang, J., Cheng, N., Zhou, S., & Xu, B. (2021). Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages (arXiv:2012.12121). arXiv. http://arxiv.org/abs/2012.12121

• Summary: The authors applied the pre-trained wav2vec2.0 model to low-resource speech recognition across six languages. Despite being pre-trained on a different domain, wav2vec2.0 could effectively adapt when fine-tuned on limited transcribed speech, even outperforming supervised pre-training approaches. Using coarser modeling units like subwords/characters worked better than finer units like phonemes/letters. Critically, self-supervised pre-training on large unlabeled data enabled wav2vec2.0 to learn robust speech representations that transferred well across languages and domains, showcasing its impressive potential for tackling low-resource speech tasks.
• RQ: Can the pre-trained wav2vec2.0 model, which was trained on English audiobook data, be effectively applied to low-resource speech recognition tasks in various languages and real-world spoken scenarios?
• Hypothesis: The self-supervised pre-training of wav2vec2.0 allows it to learn general acoustic representations that can be adapted to different languages and domains, even with limited transcribed data.
• Conclusion: The experiments demonstrate that wav2vec2.0 can achieve significant performance improvements on low-resource speech recognition tasks across six languages (Arabic, English, Mandarin, Japanese, German, and Spanish) compared to previous methods. The largest gain of 52.4% was observed for English, likely due to the pre-training data being in English. Using coarser-grained modeling units like subwords or characters generally performed better than finer-grained units like phones or letters.
• Critical observations:
  1. Self-supervised pre-training on a large amount of unlabeled data from other languages can be more effective than supervised pre-training on limited target language data.
  2. The encoder-decoder structure did not perform well in low-resource scenarios, possibly due to the decoder's inability to generalize from sparse transcriptions.
  3. External language models provided significant performance gains across all languages, model sizes, and modeling units.

• Relevance: This research highlights the potential of self-supervised pre-trained models like wav2vec2.0 to alleviate the data scarcity problem in low-resource speech recognition tasks. It demonstrates the model's ability to adapt to various languages and spoken domains, even when pre-trained on data from a different domain (audiobooks). The findings suggest that large-scale self-supervised pre-training can learn robust acoustic representations that can be effectively transferred to downstream tasks with limited data.

Thomas, B., Kessler, S., & Karout, S. (2022). Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition (arXiv:2202.03218). arXiv. http://arxiv.org/abs/2202.03218

• Summary: In this paper the authors applied adapter modules to a pre-trained wav2vec 2.0 model in order to perform downstream ASR tasks such as multilingual speech recognition. Compared with full fine-tuning, inserting adapters shows benefits of reducing the number of parameters and increasing the scalability of the model.
• RQ: The authors asked if applying adapters on self-supervised ASR models would show the same benefits as in an NLP model.
• Hypothesis: The authors hypothesized that the wav2vec 2.0 model tuned with adapter modules would be able to perform downstream tasks with little performance degradation.
• Conclusion: Self-supervised speech models can be utilized in a more parameter-efficient manner without sacrificing performance. The monolingual model such as wav2vec 2.0  can be successfully adapted to a multilingual ASR model. The multilingual model that the authors trained themselves also demonstrated capabilities to recognize English or French.
• Critical observations:
  • Adapters perform slightly worse than fine-tuning on English ASR.
  • French ASR saw a slight performance increase using adapters.
  • Multilingual pre-trained models using adapters also get close performance as in fine-tuning.
  • Adapters add only a small number of additional parameters per task.

• Relevance: This paper is the first paper that applies adapters on self-supervised ASR models. It provides insight on how adapters can be used as a quicker and computationally inexpensive method to tune the model for downstream tasks and multi-tasks. It is highly relevant to low-resource ASR because low-resource languages usually have less training data and are easy to overfit with a full fine-tuning approach. Adapter approach can prevent tuning model from overfitting.

Schultz, B.G., Tarigoppula, V.S.A., Noffs, G. et al. Automatic speech recognition in neurodegenerative disease. Int J Speech Technol 24, 771–779 (2021). https://doi-org.proxy-ub.rug.nl/10.1007/s10772-021-09836-w

• Summary: The paper evaluates the performance of three state-of-the-art automatic speech recognition (ASR) platforms (Amazon Web Services, Google Cloud, and IBM Watson) on speech from individuals with neurodegenerative diseases (multiple sclerosis and Friedreich's ataxia) and healthy controls.
• RQ: How well do commercial ASR systems perform on dysarthric speech from individuals with neurodegenerative diseases compared to healthy speech?
• Hypothesis: ASR accuracy will be lower for dysarthric speech from neurodegenerative disease groups compared to healthy controls, and accuracy will decline with increased disease severity and duration.
• Conclusion: ASR accuracy was significantly higher for healthy controls than clinical groups, and higher for multiple sclerosis compared to Friedreich's ataxia. Amazon Web Services and Google Cloud outperformed IBM Watson. Accuracy decreased with increased disease duration for Friedreich's ataxia but not multiple sclerosis. Age and sex did not significantly affect ASR accuracy.
• Critical observations:
  • ASR faces challenges in recognizing dysarthric speech from neurodegenerative diseases.
  • Accuracy declines as consecutive words increase, irrespective of speech impairment.
  • Severity of speech impairment, as indicated by disease type and duration, negatively impacts ASR accuracy.
• Relevance: The theme focuses on low-resource ASR for underrepresented languages. While this study does not directly address low-resource languages, it highlights the challenges ASR systems face in recognizing atypical speech patterns, which is relevant for low-resource languages with diverse speaker populations and dialects. Improving ASR performance on dysarthric speech could inform techniques for handling speech variability in low-resource settings.

Vásquez-Correa, J. C., Rios-Urrego, C. D., Arias-Vergara, T., Schuster, M., Rusz, J., Nöth, E., & Orozco-Arroyave, J. R. (2021). Transfer learning helps to improve the accuracy to classify patients with different speech disorders in different languages. Pattern Recognition Letters, 150, 272–279. https://doi.org/10.1016/j.patrec.2021.04.011

• Summary: The paper proposes using transfer learning with convolutional neural networks (CNNs) to classify pathological speech from patients with neurodegenerative disorders like Parkinson's disease (PD) and Huntington's disease (HD). Time-frequency representations of voice onset/offset segments are used as input to the CNNs. Two transfer learning scenarios are explored: 1) transferring a model trained on one language to classify patients speaking a different language, and 2) transferring a model trained on one disorder (e.g. PD) to classify patients with a different disorder (e.g. HD).
• RQ: Can transfer learning improve the accuracy of CNN models for classifying pathological speech across different languages and disorders?
• Hypothesis: Transferring knowledge from a base model trained on one language/disorder to a target model for a different language/disorder can improve classification accuracy when there is limited data for the target task.
• Conclusion: The results suggest transfer learning can improve target model accuracy, but only when the base model is sufficiently accurate. Transferring between similar tasks (e.g. different languages) works better than transferring between very different tasks (e.g. different disorders).
• Critical observations:
  • Accuracies ranged from 70-89% across languages without transfer learning
  • Transferring between languages improved accuracy in some cases (e.g. Spanish -> German improved over training on German alone)
  • Transferring between very different disorders like PD and HD did not improve over training directly on the target disorder
• Relevance: The paper does not directly address low-resource ASR, but instead focuses on pathological speech classification. However, some insights around transfer learning across languages could potentially be adapted to low-resource ASR scenarios.

Synthesis

In summary, these articles investigate various approaches to enhancing Automatic Speech Recognition (ASR) systems, particularly focusing on low-resource languages, accent variations and model compression. SpecAugment demonstrates effectiveness in speaker verification tasks across different languages, while Google USM explores leveraging unlabelled data for multilingual ASR. Additionally, data augmentation techniques are shown to mitigate biases against non-native accents in Flemish ASR systems. Self-supervised speech models like wav2vec 2.0 and adapter transfer techniques are also explored to leverage unlabeled data and efficiently adapt pre-trained models. These findings collectively underscore the importance of robust and inclusive ASR technology for diverse linguistic contexts, prompting further exploration into tailored augmentation strategies and multilingual model development to address the challenges of low-resource languages and accent diversity. The combination of transfer learning and data augmentation have also shown potential for improving ASR performance when only limited data is available, by leveraging knowledge from higher-resource languages or domains.

Contributors

Contributors: Ömer Tarik, Xinyi Ma, Cantao Su, Page Ouyang, Weixi Lai, Xueying Liu

• Artice Wang et al., 2019: Xinyi Ma
• Article Google USM: Scaling automatic speech recognition beyond 100 languages: Ömer Tarik
• Article Zhang et al., 2023: Xinyi Ma
• Article Wang, H. et al., 2023: Page Ouyang
• Article Gandhi S. et al., 2023: Page Ouyang
• Article Yang M. et al., 2023: Page Ouyang
• Article N, K. D et al., 2021: Weixi Lai
• Article Yi et al., 2021: Weixi Lai
• Article Thomas et al., 2022: Xueying Liu
• Article Automatic speech recognition in neurodegenerative disease, 2021: Cantao Su
• Article Transfer learning helps to improve the accuracy to classify patients with different speech disorders in different languages, 2021: Cantao Su
• Introduction: Ömer Tarik, Cantao Su
• Synthesis: Xinyi Ma, Cantao Su, Weixi Lai, Page Ouyang

Language-specific Text-To-Speech

Introduction

State-of-the-art Text-to-Speech systems have different performances based on the language they are developed for and trained on. We choose to focus on language-specific TTS and provide a review of state-of-the-art techniques to synthesise languages other than English. Even though this does not necessarily restrict to Low-Resourced Languages (LRLs), we decided to focus mainly on techniques developed for LRLs, and more broadly, approaches that entail the use of a limited amount of data.

The article summaries below include the topics of multilingual data strategies, TTS with phonological features, and Transfer Learning.

Article summaries

Do, P., Coler, M., Dijkstra, J., & Klabbers, E. (2021). A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages. Proc. Interspeech 2021, 16–20. doi: 10.21437/Interspeech.2021-1565

• Summary: The article provides an overview of strategies for test-to-speech for low-resource langauges, focusing on Multilingual Data strategies. More specifically, this article presents an evaluation of the results of the previous studies on LRLs TTS, an evaluation of the influence of data augmentation techniques employed on the performance of the models and the proposal of a new measure to evaluate the performance of multilingual vs. monolingual systems with different evaluation metrics, namely MultiLingual Model Effect (MLME). The performance of the strategies analysed is also checked by verifying how different factors influence it.
• RQ:
  1. Using the same limited amount of LRL data, how does the output quality of multilingual TTS models compare to that of monolingual models?
  2. What factors in the data augmentation strategy influence the effect of using multilingual TTS models on output quality, and to what extent do they affect it?
• Hypothesis: Looking at the correlations between data augmentation strategies and synthesized speech quality, tools that use multilingual data can be provided for future research in TTS for LRLs, especially regarding the efficiency of using such data.
• Conclusion: Multilingual approaches are more effective in training for LRLs. The factors that affect the performance are:
  • target language data ratio between corresponding multilingual and monolingual models;
  • target language data balance ratio over total training data
  • amount of target language data.
• Critical observations: The paper only focuses on multilingual data strategies, and justifies the choice by saying that multispeaker data are harder to collect for LRLs. Even though I understand the reasoning behind this, I believe this is not entirely true. On one hand, it is true indeed that it is harder to find many speakers for a LRLs, since oftentimes such languages are also minority languages. On the other hand, collecting multispeaker data means that each speaker can contribute with a very small amount of data and still get enough of them. This means that by adopting multispeaker TTS techniques, we don't need to record one speaker for a long time, but rather multiple speakers for a short time. This multi-speaker approach, I believe, could be used in combination with Transfer Learning to improve the results of LRLs TTS systems, even though this implies adding complezity to the pipeline.
• Relevance: The most relevant outcome of this study, especially for LRLs TTS, is that the language family is not relevant for the selection of the target-source language pair, no matter the architecture. In my opinion, the conclusions of this paper are also relevant for medium-resourced languages and in general for the synthesis of non-standard speech and for all the types of speech that are not widely covered by the research so far.

Staib, M., Teh, T. H., Torresquintero, A., Mohan, D. S. R., Foglianti, L., Lenain, R., & Gao, J. (2020). Phonological features for 0-shot multilingual speech synthesis. arXiv preprint arXiv:2008.04107.

• Summary: This article primarily aims to utilize a limited set of phonological features (PF), derived from the International Phonetic Alphabet (IPA), for achieving 0-shot speech synthesis and code-switching within a monolingual model. Specifically, the study selects Tacotron 2 as the baseline for comparison against methods of random initialization (RANDOM), manual mapping (MANUAL), and the PF-based approach (AUTO) proposed in this work. The conclusion drawn is that the speech generated using the AUTO method is more comprehensible.
• RQ: The research question of this paper explores whether phonological features (PF) can facilitate speech synthesis for languages not seen during training. Additionally, it examines whether PF can facilitate code-switched speech synthesis.
• Hypothesis: The hypothesis of the article is that phonological features (PFs), derived from the International Phonetic Alphabet (IPA), can enable 0-shot text-to-speech (TTS) synthesis and code-switching in languages that are not seen during training, even within monolingual models.
• Conclusion: The conclusion of the article is that by replacing the character input in Tacotron 2 with phonological features, a model topology can be created that is language-independent and allows for the automatic approximation of sounds unseen in training. The study shows that phonological features (PFs) can not only facilitate zero-shot speech synthesis in untrained languages within a small multilingual or even a monolingual model but also facilitate the synthesis of sounds that completely unseen in training. This suggests potential applications in code-switching and TTS for low-resource languages.
• Critical observations: This article mainly addresses the problem of 0-shot speech synthesis and code switching by using phonological features (PF). The significant advantage of this method is that it can reduce the amount of data for training multi-language speech synthesis models, and it is very helpful in low-resource languages ​​and code-switched TTS. But PF may not capture all the differences of a language, especially for those with unique phonetic and phonological features, and the selected PFs might not adequately represent these languages. In addition, this article focuses more on generating understandable speech and may ignore the importance of features such as prosody.
• Relevance: This paper is mainly related to the fileds of cross-language speech synthesis and code switching speech synthesis. Some other studies have also proposed to find a unified representation (such as Unicode) to replace phoneme or text to achieve cross-language synthesis, but the PF proposed in this paper may be A better choice because these features retain speech features to a certain extent and help the model learn better.

Do, P., Coler, M., Dijkstra, J., & Klabbers, E. (2023). Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection. arXiv preprint arXiv:2306.12040.

• Summary: This paper compares two methods in TTS for low-resource languages: PHOIBLE-based phone mapping and phonological features input. Various languages are tested to see how these methods work across different languages. The findings show that both methods improve speech quality, with phonological features performing better. The study also examines two criteria for choosing source languages: Angular Similarity of Phone Frequencies (ASPF) and language family tree distance. ASPF is found effective, especially with phone-based input, while the language distance criterion does not yield expected results.
• RQ: The paper aims to explore how to most effectively deal with the input mismatch between languages and how to select the best source language to improve output quality in TTS for low-resource languages.
• Hypothesis:
  1. Transfer learning using PHOIBLE-based phone mapping and phonological feature inputs can improve TTS output quality for low-resource languages.
  2. Angular Similarity of Phone Frequencies (ASPF) is an effective criterion for selecting source languages, more so than traditional broad language family classification.
• Conclusion:
  1. Both phone mapping and feature inputs can enhance output quality, with feature inputs showing better performance, although the effectiveness depends on the specific language pairing.
  2. ASPF is effective in selecting source languages, especially when using label-based phone inputs, while the distance based on the language family tree does not work as expected.
• Critical observations:
  1. Although ASPF is effective in some cases, its effectiveness is not universal across all language combinations, indicating the need for further research to understand influencing factors.
  2. The unexpected results with the language family tree distance suggest that there might be unidentified factors at play, necessitating further investigation.
• Relevance: This research is significant for the development of TTS technology for low-resource languages, especially in offering new insights into source language selection and handling input mismatches between languages. Moreover, the proposed methods are important for the multilingual applicability and scalability of speech technologies.

Wells D, Richmond K. Cross-lingual transfer of phonological features for low-resource speech synthesis[C]//Proceedings of the 11th Speech Synthesis Workshop, Budapest, Hungary. 2021: 160-165.

• Summary: In this paper, researchers compare two methods: fine-tuning phonemic representations and using phonological features. They used SPE-style phonological features, offering a binary representation of phonemes, which helps describe and analyze speech patterns in English and German. The study discovers that even with limited target language data, fine-tuning can generate speech comparable to models trained from scratch. Using phonological features slightly improves naturalness ratings compared to using phonemes alone. These findings highlight the practical benefits of phonological features in improving TTS output quality across languages.
• RQ: Does the use of different input representations (phonemes and phonological features) affect the naturalness of synthesized speech in text-to-speech synthesis using cross-lingual transfer learning?
• Hypothesis: In cross-lingual transfer learning for text-to-speech synthesis, the use of different input representations (phonemes and phonological features) affects the naturalness of synthesized speech.
• Conclusion: The study confirmed the effectiveness of cross-lingual fine-tuning for training synthetic voices with limited target language data. Phonological features were found to offer practical benefits over phonemes in terms of parameter sharing during transfer learning.
• Critical observations: There was a slight improvement in naturalness ratings when using PFs over phonemes. Future research may explore multilingual grapheme-to-phoneme systems and utilize additional linguistic resources to enhance low-resource pipelines for text-to-speech synthesis
• Relevance: Phonological features were found to offer practical benefits over phonemes in terms of parameter sharing during transfer learning, which can be applied greatly in LRLs TTS.

Synthesis

To summarize, text-to-speech research in recent years has explored multilingual data strategies, phonological features, and transfer learning methods to enhance its performance, especially for low-resource languages.

Based on the studies reported above, multilingual models outperform monolingual ones, showing promise in improving the voice quality with limited data. Moreover, phonological features facilitate zero-shot synthesis and code-switching, benefiting LRLs and cross-language applications. Transfer learning methods like PHOIBLE-based phone mapping and phonological feature inputs improve output quality, with ASPF effective for source language selection. Finally, fine-tuning phonological representations enhances speech naturalness, suggesting the potential for multilingual g2p systems.

These findings emphasise innovative approaches' importance in advancing TTS, with a specific focus on LRLs, offering insights into effective strategies and criteria for synthesis quality and scalability.

Contributors

• Article Do, et al. (2021) 'A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages': Alice Vanni
• Article Staib et al. (2020) 'Phonological features for 0-shot multilingual speech synthesis': Wang Yinqiu
• Article Do, et al. (2023) 'Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection': Annie Zhou
• Article Wells D, et al. (2021) 'Richmond K. Cross-lingual transfer of phonological features for low-resource speech synthesis': Ding
• Introduction: All
• Synthesis: All

Theme: TTS naturalness

Introduction

TTS systems have significantly advanced over time, achieving remarkable intelligibility and near-human naturalness in synthetic voices through deep learning advancements. However, the naturalness of synthetic voice remains limited to sentences, and lacks the expressivity found in human conversation such as appropriate emotion, prosody and style. Despite these limitations, natural TTS, particularly expressive speech synthesis, plays a crucial role in achieving human-like speech and enhancing the engagement of synthesized speech. Moreover, it facilitates the broader adoption of TTS technology across various domains within the field of speech technology. In this context, our group focuses on the theme of TTS naturalness with two interconnected subthemes: exploring advanced models and relevant theories. By addressing these subthemes, we aim to provide a comprehensive overview of the current state-of-the-art in TTS naturalness.

Article summaries

Subtheme 1: State-of-the-art Models

Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., Soong, F., Qin, T., Zhao, S., & Liu, T.-Y. (2022). NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality. arXiv preprint arXiv:2205.04421.

• Summary: NaturalSpeech proposes a system for converting text to speech (TTS) that achieves human-level quality. It leverages a variational autoencoder (VAE) to bridge the gap between text and speech waveforms.
• RQ (Research Question): Can a TTS system achieve speech quality indistinguishable from humans?
• Hypothesis: By incorporating a VAE and specific techniques to improve the model's understanding of text and speech features, NaturalSpeech can generate speech indistinguishable from humans.
• Conclusion: The paper argues that NaturalSpeech achieves human-level speech quality based on statistical measures (MOS and CMOS) in human evaluations.
• Critical Observations: The evaluation relies on subjective human ratings, which might be influenced by factors beyond speech quality.The research focuses on a single benchmark dataset, limiting generalizability.The paper doesn't explore how NaturalSpeech performs on diverse speaking styles or accents.

• Relevance: This is related to my study because it provides a definition of human-level quality, and this particular model has achieved the highest Mean Opinion Score (MOS) recorded thus far. Hence, I am considering using this model as a basis for my study.

Kong, J., Kim, J., & Bae, J. (2020). Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33, 17022-17033.

• Summary: This article introduces HiFi-GAN, a model that can efficiently synthesize high-quality speech audio. HiFi-GAN consists of a generator and two discriminators: multi-scale discriminator and multi-period discriminator. Improve training stability and model performance by adversarially training the generator and discriminator and using two additional loss functions.

• RQ:Can HiFi-GAN effectively synthesize high-quality speech audio with computational efficiency comparable to human-level synthesis, while also demonstrating generalization across speakers and adaptability to various configurations?
• Hypothesis:By leveraging the characteristic patterns of speech audio and designing a discriminator to capture these patterns effectively, it is possible to develop a speech synthesis model, HiFi-GAN, that outperforms existing models in terms of synthesis quality and speed.
• Conclusion:HiFi-GAN significantly advances speech synthesis by efficiently generating high-quality audio, surpassing existing models in both synthesis quality and speed. By leveraging speech audio patterns and a carefully designed discriminator, this model demonstrates robustness across various scenarios, including unseen speakers and noisy inputs, while offering potential for on-device natural speech synthesis with low latency and memory requirements. Additionally, the flexibility of generator configurations enhances adaptability without the need for extensive hyper-parameter search.
• Critical observations:Due to the wide application of HiFi-GAN technology in the field of speech synthesis, there may be some ethical or social impacts, including concerns related to voice cloning, privacy and false information.
• Relevance:This paper is closely related to the topic of non-language-specific text-to-speech, as it demonstrates a breakthrough in HiFi-GAN models in synthesizing high-quality speech, with generalization capabilities, and the ability to handle inputs of different languages and speaking styles.

Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., ... & Zhao, Z. (2023, July). Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning (pp. 13916-13932). PMLR.

• Summary: The article introduces "Make-An-Audio," a system utilizing a prompt-enhanced diffusion model for TTS generation, aiming to improve the naturalness and expressiveness of synthesized audio.
• RQ: How does the model improve the naturalness of TTS?
• Hypothesis: By introducing pseudo prompt enhancement and spectrogram autoencoders, the model can effectively utilize unsupervised language-free data and higher-level semantic understanding to enhance the naturalness and expressiveness of speech synthesis.
• Conclusion: "Make-An-Audio" successfully enhances the naturalness and expressiveness of speech synthesis, achieving state-of-the-art performance in evaluations.
• Critical observations: The performance of "Make-An-Audio" is still partly dependent on extensive data and complex model training. In addition, there is still space for improvement in expressing the emotions and rhythms of human conversations.
• Relevance: The "Make-An-Audio" system presented in the paper offers an effective solution to the limitations in naturalness and expressiveness currently faced by TTS

Subtheme 2: State-of-the-art Theories

Noufi, C., May, L., & Berger, J. (2023). Context, Perception, Production: A Model of Vocal Persona. PsyArXiv. July, 28.

• Summary: This article introduces a contextualized production-perception model of vocal persona, developed through qualitative analysis of interviews with voice and performance experts. It emphasizes the influence of context on an individual's vocal expression, reflecting the intricacies of human communication.
• RQ: What is the relationship between context, vocal expression, and identity?
• Hypothesis: It is a qualititative study and does not hve a formulated hypothesis. Instead of attempting to falsify a hypothesis as in most quantitative studies, it explores answers to the research question through thematic analysis.
• Conclusion: Speakers actively select different vocal personas and adjust relevant vocal expressions in response to the surrounding context, facilitating a transition in the perception of persona.
• Critical observations: The proposal of the vocal persona model and general conclusions are based on interviews with 21 voice and performance experts, which may have limitations in terms of subjective bias and generalizability beyond this specific context.
• Relevance: This study underscores the necessity for speakers to adapt their speaking styles to accommodate different social contexts, highlighting the significance of context in vocal expression. It proposes the incorporation of vocal persona into expressive vocal synthesis with a three-spoke model and a framework for persona-guided vocalization, enriching the framework of TTS naturalness and expressiveness.

Vainer, J., & Dušek, O. (2020). Speedyspeech: Efficient neural speech synthesis. arXiv preprint arXiv:2008.03802.

• Summary: This paper introduces a novel student-teacher network architecture called "SpeedySpeech" for fast and high-quality neural speech synthesis. The system is designed to enable faster-than-real-time speech synthesis while requiring minimal computing resources, and deliver audio quality that is superior to existing models such as the Tacotron 2. The model uses the teacher network for duration extraction, the student network for spectrogram synthesis, and combines it with the MelGAN vocoder to output high-quality audio. The training process is efficient and can be completed in less than 40 hours on a single 8GB GPU.
• RQ: How can we develop a neural speech synthesis system that does not require extensive computing resources while maintaining fast training times, fast inference, and high-quality audio output?
• Hypothesis: Assuming a student-teacher network architecture with simplified convolutional blocks and only a single attention layer in the teacher model, it is possible to surpass existing models in terms of training efficiency and audio quality while maintaining fast inference.
• Conclusion: The proposed SpeedySpeech model successfully achieves its goals by demonstrating that self-attention layers are not necessary for high-quality audio generation and that simpler, fully convolutional methods enable a more efficient training process and faster synthesis. The model's speech quality score is significantly higher than Tacotron 2, and it can be trained efficiently on a single GPU and even run in real time on the CPU.
• Critical observations: The article proposes ways to address the trade-off between training efficiency and audio quality in neural speech synthesis. By using only a single attention layer in the teacher model and eliminating sequence generation in the student network, the authors achieve important simplifications that increase model efficiency. In the model evaluation, the authors comprehensively considered objective indicators (such as MAE, SSIM) and subjective listening tests to provide a comprehensive assessment of model performance.
• Relevance: This speech synthesis model has applications in many fields, including virtual assistants, machine translation, etc. The SpeedySpeech model can synthesize speech in real time on moderate hardware, making it particularly suitable for deployment in resource-constrained environments. Additionally, the focus on efficiency and quality sets new benchmarks for future research and development in this area.

Peiró-Lilja, A., & Farrús, M. (2020). Naturalness Enhancement with Linguistic Information in End-to-End TTS Using Unsupervised Parallel Encoding. Interspeech 2020, 3994–3998. https://doi.org/10.21437/Interspeech.2020-1788

• Summary: The paper explores enhancing the naturalness of synthesized speech in E2E-TTS systems by incorporating linguistic features like POS tags and punctuation into the Tacotron 2 model, aiming to improve prosody to resemble human speech more closely.
• RQ: How can linguistic information be integrated into the Tacotron 2 system to improve the naturalness of synthesized speech prosody?
• Hypothesis: The hypothesis is that by embedding POS tags and punctuation locations as additional linguistic features into the Tacotron 2 system, the synthesized speech will exhibit improved naturalness and prosody, making it more similar to human speech.
• Conclusion: The study concludes that the incorporation of linguistic features through a parallel encoder significantly improves the naturalness of synthesized speech. The authors proposed two different architectures for the parallel encoder: one based on convolutional and recurrent layers (2DConv+BiLSTM) and another composed of bidirectional recurrent and linear layers (BiGRU+Linear). Both architectures aimed to process the binary matrix representing POS tags and punctuation locations. The results from objective tests and perceptual evaluations indicated that the model with the 2DConv+BiLSTM parallel encoder performed the best in terms of naturalness, as it more closely matched human pitch contours and overall speech quality.
• Critical observations: Critically, the paper notes that while both parallel encoder architectures showed improvements over the Tacotron 2 baseline, the 2DConv+BiLSTM version provided better results in terms of naturalness. However, it also introduced a slight increase in Mel Cepstral Distortion (MCD), suggesting a trade-off between naturalness and certain acoustic quality metrics. The BiGRU+Linear model, despite being lighter and faster, underperformed in perceptual tests, possibly due to its reduced complexity and higher cepstral distortion.
• Relevance: The findings of this research are relevant for the development of more natural and human-like E2E-TTS systems, which have applications in various domains such as automatic dialogue systems, storytelling, and voice assistants. By enhancing the prosody of synthesized speech, these systems can provide more engaging and realistic interactions, improving user experience and accessibility. Furthermore, the study contributes to the broader field of speech synthesis by demonstrating the potential of unsupervised parallel encoding of linguistic features to improve speech naturalness.

Cai, X., Dai, D., Wu, Z., Li, X., Li, J., & Meng, H. (2021). Emotion Controllable Speech Synthesis Using Emotion-Unlabeled Dataset with the Assistance of Cross-Domain Speech Emotion Recognition. ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5734–5738. https://doi.org/10.1109/ICASSP39728.2021.9413907

• Summary: This article proposes an approach for emotional TTS synthesis on a dataset without emotion labels, using a cross-domain speech emotion recognition model and an emotional TTS model, hoping to express similar results in specific emotional expressiveness and speech quality compared to models with emotion labels.
• RQ: Can we use the achievements and features of SER to solce the problem of lack of emotion-annotated dataset for emotional TTS?
• Hypothesis: By performing mean opinion score evaluations and emotion recognition perception evaluation in 4 emotion categories and 2 polarities of emotion dimensions, our GST-based model can generate speech with expected emotions, while trained on a fully emotion-unlabeled dataset.
• Conclusion: Through comparing their created cross-domain model with a baseline model, they found that both their 4-categorical model and 2-dimensional model almost achieve as good speech quality as the baseline system, with higher p-values than the significance level of 0.05, indicating no significant differences. Furthermore, they found that both categorical models, one trained on the utterances with highest posterior (top-K scheme) and one trained on the full set of audio, revealed an overall higher accuracy than the baseline model, at 78.75% and 49.25%, respectfully, compared to the baseline 36.75%, which indicated their cross-domain model and top-K scheme were effective in emotional expressiveness.
• Critical observations: The choice to use a top-K scheme, as described earlier, is interesting to offset the number of mispredictions made by the SER model, as the SER model is far less reliable that humans. By choosing to use the more reliable audio set, there could be an argument that their choice could inflate their results. However, taking this into account, they did in fact train the model on the full, unaltered, set of audio and still returned a higher accuracy than the baseline model. The fact that this shows promise in returning accurte and sufficient quality in emotional speech synthesis with unlabeled emotion datasets brings promise to a possible increase in speed and efficiency for training other models.
• Relevance: The proposed approach, in the authors' words, greatly reduces the threshold of emotional synthesis in regard to amotion-annotated data, reducing the time, cost, and relevant quality of the speech data needed for emotional TTS systems.

Synthesis

From the articles on non-language-specific text-to-speech (TTS) synthesis highlights several emerging trends and debates within the field of voice technology. The articles reviewed contribute to a comprehensive understanding of the state-of-the-art in TTS naturalness, spanning advanced models and theories that aim to bridge the gap between synthetic and human speech.

Emerging Trends:

1. Advancements in Model Architecture: A significant trend is the development of advanced TTS models, such as NaturalSpeech, Make-an-Audio, HiFi-GAN, and SpeedySpeech, which leverage innovative techniques like variational autoencoders, prompt-enhanced diffusion models, adversarial training, and efficient network architectures. These models aim to improve the naturalness and expressivity of synthetic speech, achieving closer approximation to human speech quality.

2. Integration of Linguistic and Emotional Information: There is a growing emphasis on incorporating linguistic features and emotional expressivity into TTS systems. Studies like the one by Peiró-Lilja & Farrús, and Cai et al. demonstrate the potential of enhancing speech naturalness and emotional expressivity by embedding linguistic cues and leveraging emotion-unlabeled datasets with cross-domain speech emotion recognition models. This approach points to a nuanced understanding of speech production, where prosody, context, and emotional tone play crucial roles.

3. Exploration of Vocal Persona and Contextual Factors: The study by Noufi, May, & Berger introduces the concept of vocal persona, highlighting the influence of context on vocal expression and identity. This reflects an acknowledgment of the complexity of human speech, where individuals adapt their vocal style to different social contexts. Integrating such contextual and persona-based nuances into TTS systems could lead to more sophisticated and contextually aware speech synthesis.

Debates:

Quality vs. Complexity: Despite advancements, a recurring challenge is the trade-off between improving speech quality and managing the complexity and computational demands of TTS models. Models like HiFi-GAN and SpeedySpeech address this by optimizing for efficiency and fidelity, yet questions remain about the balance between model simplicity and the ability to capture the rich variability of human speech.

In conclusion, the field of TTS is witnessing rapid advancements and facing complex challenges. The synthesis of findings from the reviewed articles underscores the importance of multidisciplinary approaches that integrate technical innovations with insights from linguistics and psychology to advance towards more natural, expressive, and ethically developed TTS technologies.

Contributors

Contributors: A list of contributors by contribution

• Article Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models: Yilan Wei
• Article Context, Perception, Production: A Model of Vocal Persona: Chenyi Lin
• Article NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality: Yi Lei
• Article HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis: Yanhua, Liao
• Article Naturalness Enhancement with Linguistic Information in End-to-End TTS Using Unsupervised Parallel Encoding: Jingxuan Yue
• Article Emotion Controllable Speech Synthesis Using Emotion-Unlabeled Dataset with the Assistance of Cross-Domain Speech Emotion Recognition: Jocomin Galarneau
• Article SpeedySpeech: Efficient Neural Speech Synthesis: Weihao Jiang
• Introduction: Chenyi Lin
• Synthesis: Yi Lei

Theme: ASR

Introduction

The rapid evolution of Automatic Speech Recognition (ASR) technology has been a cornerstone in advancing how humans interact with machines, propelling us towards more seamless and intuitive communication avenues. The focus on ASR technology underscores its pivotal role across a myriad of applications, from enhancing accessibility and providing robust customer support solutions to creating immersive interactive entertainment experiences. Among the most intriguing challenges in this domain is the recognition and interpretation of complex human sentiments such as sarcasm and humor. These nuanced forms of expression, deeply embedded in human language, present unique challenges for ASR systems due to their reliance on contextual cues, background knowledge, and the subtle modulations in tone that conventional speech recognition systems often miss. Our exploration is driven by the imperative to bridge this gap, aiming to refine ASR technology's ability to discern and process these complex sentiments.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Wang S, Yang C H H, Wu J, et al. Can whisper perform speech-based in-context learning[J]. arXiv preprint arXiv:2309.07081, 2023.

• Summary: The study investigates Whisper ASR models' in-context learning capabilities and proposes a novel SICL method for test-time adaptation without gradient descent, achieving significant WER reductions.
• RQ: The research explores whether Whisper models can perform speech-based in-context learning and how to leverage in-context examples for test-time adaptation efficiently.
• Hypothesis: The hypothesis is that Whisper models can adapt at test time using SICL with context examples from specific dialects or speakers.
• Conclusion: SICL significantly improves ASR performance for Chinese dialects without gradient descent, with k-NN enhancing SICL's efficiency.
• Critical observations: Correct LID settings and k-NN example selection improve Whisper's inference, with language-level adaptation outperforming speaker-level adaptation.
• Relevance: The study is relevant for understanding and enhancing the application of large pre-trained models in automatic speech recognition and dialect adaptation.

Sungjoo Ahn and Hanseok Ko. “Background Noise Reduction via Dual-Channel Scheme for Speech Recognition in Vehicular Environment.” IEEE Transactions on Consumer Electronics 51, no. 1 (February 2005): 22–27. https://doi.org/10.1109/TCE.2005.1405694.

• Summary: The paper proposes a dual-channel noise reduction method aimed at enhancing speech recognition systems within vehicular environments, characterized by significant noise challenges. The authors argue that existing single-channel methods fall short in effectively improving speech recognition performance due to inherent noise complexities in vehicles. The proposed method leverages a high-pass filter combined with an eigen-decomposition front-end processing technique, tested against real multi-channel vehicular corpus. Experimental results indicated a notable improvement in speech recognition performance using various microphone arrangements, showcasing the superiority of the dual-channel approach over traditional single-channel methods.
• RQ: How can the performance of speech recognition systems in vehicular environments be improved through a dual-channel noise reduction scheme?
• Hypothesis: The paper hypothesizes that employing a dual-channel noise reduction scheme, which integrates a high-pass filter with eigen-decomposition front-end processing, can significantly enhance speech recognition performance in noisy vehicular environments by effectively distinguishing speech from background noise.
• Conclusion: Authors concluded that their dual-channel noise reduction method, especially when augmented with a high-pass filter and enhanced eigen-decomposition processing, substantially improves speech recognition accuracy in vehicular settings. The method outperformed standard single-channel noise reduction approaches and showed considerable promise in overcoming the challenges posed by vehicular background noise, thereby validating the hypothesis.
• Critical observations: The study successfully demonstrates the effectiveness of a dual-channel approach in a challenging noise environment. However, the practical deployment of such systems, including the economic implications and the adaptability across different vehicle models and noise conditions, remains less explored. Additionally, while the study marks a significant improvement over existing methods, the scalability of this approach in terms of computational demand and real-time processing capabilities could benefit from further investigation.
• Relevance: This thesis is relevant to the topics in enhancing speech recognition technology area. The innovative approach of combining a dual-channel noise reduction scheme with a high-pass filter and eigen-decomposition method provides a substantial leap forward in developing more reliable and efficient speech recognition systems.

Zhang, Wangyou, and Yanmin Qian. “Weakly-Supervised Speech Pre-Training: A Case Study on Target Speech Recognition.” arXiv, June 29, 2023. http://arxiv.org/abs/2305.16286.

• Summary: This study introduces a new way to teach computers to understand speech by focusing on one person's voice in a noisy place, like when many people talk at once. This method, called TS-HuBERT, uses extra information about the speaker's voice to improve speech recognition, especially in challenging situations with lots of background noise. Tests showed that TS-HuBERT does a better job than other similar methods, making it a promising approach for better understanding speech in noisy environments.
• RQ: Can we use extra information about who is speaking to help computers better recognize speech in noisy settings?
• Hypothesis: By using additional information about the speaker, the TS-HuBERT method can focus on the target speaker's voice more effectively, even when other voices or noises are present.
• Conclusion: TS-HuBERT improves speech recognition by focusing on the target speaker's voice, outperforming other current methods. This approach is particularly useful for recognizing speech in noisy places where many people are talking at once.
• Critical observations:
  • TS-HuBERT can be adjusted to different speech recognition tasks, showing its versatility.
  • Although it needs extra information about the speaker's voice, this method greatly enhances the computer's ability to focus on and understand the target speaker in noisy situations.
  • There is still room for improvement, especially in very noisy environments, indicating potential areas for future research.
• Relevance: This study is directly relevant to the topic to help computers understand speech better in challenging environments, like when many people are talking at the same time. By focusing on a specific speaker's voice, TS-HuBERT could make speech recognition technology more effective in real-world situations.

Bae, S., Kim, J.-W., Cho, W.-Y., Baek, H., Son, S., Lee, B., Ha, C., Tae, K., Kim, S., & Yun, S.-Y. (2023). Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification. Retrieved from https://arxiv.org/abs/2305.14032v4

Summary: The study introduces a novel approach for respiratory sound classification, leveraging a pretrained Audio Spectrogram Transformer (AST) model, alongside a new Patch-Mix augmentation technique and Patch-Mix Contrastive Learning. These methods are designed to address the challenges of medical data scarcity and enhance model performance on the ICBHI dataset. The approach sets a new state-of-the-art performance benchmark, improving the classification Score by 4.08% over previous methods.

• RQ: Can a pretrained Audio Spectrogram Transformer (AST) model, combined with Patch-Mix augmentation and Patch-Mix Contrastive Learning, effectively improve respiratory sound classification, especially in the context of the ICBHI dataset?
• Hypothesis: The hypothesis posits that leveraging a pretrained AST model, which has been trained on large-scale visual and audio datasets, can be effectively generalized to respiratory sound classification tasks. Additionally, it suggests that the introduction of Patch-Mix augmentation and Patch-Mix Contrastive Learning can further enhance model performance by addressing the scarcity of medical data and the challenges of leveraging such data for deep learning models.
• Conclusion: The study concludes that the proposed approach, combining a pretrained AST model with Patch-Mix augmentation and Patch-Mix Contrastive Learning, significantly enhances respiratory sound classification. This method achieved state-of-the-art performance on the ICBHI dataset, demonstrating the effectiveness of the proposed techniques in improving classification accuracy in the face of limited medical data availability and complex data characteristics.
• Critical observations:
  • Pre-training on both visual and audio domains using the AST model shows substantial improvements in generalizing to respiratory sound classification tasks.
  • The Patch-Mix augmentation technique, which randomly mixes patches between different samples, and the Patch-Mix Contrastive Learning method, which distinguishes mixed representations in the latent space, effectively mitigate the overfitting issue and enhance model robustness.
  • The study's methodology offers a significant performance increase, demonstrating the potential of attention-based models and contrastive learning in medical sound classification.
• Relevance: This research holds relevance to Automatic Speech Recognition (ASR) by showcasing the utility of attention-based models like the AST in capturing long-range dependencies in audio data. The techniques developed for respiratory sound classification, particularly the effective use of pretrained models and innovative augmentation strategies, can inform similar challenges in ASR, including dealing with limited training data and enhancing model generalization across diverse audio inputs.

Gairola1, S., Tom, F., Kwatra1, N., & Jain1, M. (2021). RESPIRENET: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting. Retrieved from https://arxiv.org/abs/2011.00196v2

• Summary: The study introduces RespireNet, a CNN-based model for classifying respiratory sounds, particularly focusing on addressing the challenge posed by the small size of the largest available respiratory dataset, ICBHI, which consists of only 6,898 breathing cycles. The study proposes a suite of novel techniques including device-specific fine-tuning, concatenation-based augmentation, blank region clipping, and smart padding to efficiently utilize this small dataset. Extensive evaluation on the ICBHI dataset demonstrates significant improvements over state-of-the-art results for 4-class classification by 2.2%.
• RQ: Can a simple CNN-based model, when combined with specific data utilization techniques, accurately classify respiratory sounds from a limited-sized dataset, overcoming the challenges of data scarcity and variability?
• Hypothesis: The study hypothesizes that even with a small dataset, a simple network architecture, if supplemented with innovative techniques for data utilization and augmentation, can accurately classify respiratory sounds. These techniques include addressing dataset characteristics such as device variability, class imbalance, and varying audio lengths that traditionally inhibit effective DNN training.
• Conclusion: RespireNet, along with the proposed data utilization techniques, significantly improves the accuracy of respiratory sound classification, achieving new state-of-the-art performance on the ICBHI dataset for both 2-class and 4-class classification tasks. The study concludes that focusing on efficient data utilization and addressing specific dataset characteristics can compensate for the limitations posed by small-sized datasets.
• Critical observations:
  1. Transfer learning from pre-trained ImageNet models proves beneficial, suggesting that even unrelated domain knowledge can improve model performance.
  2. Concatenation-based augmentation effectively addresses class imbalance, significantly improving classification of underrepresented classes.
  3. Device-specific fine-tuning is essential for generalizing across different recording devices, highlighting the impact of hardware variability on model performance.
  4. Techniques like smart padding and blank region clipping are crucial for dealing with variable-length audio samples and irrelevant frequency regions, respectively, ensuring the model focuses on relevant features.
• Relevance: The challenges and solutions presented in this study have direct implications for ASR, especially in scenarios where data is scarce or highly variable. Techniques such as smart data augmentation, device-specific adjustments, and focusing on relevant audio features can be applied to improve ASR systems' robustness and accuracy in diverse conditions. Furthermore, the emphasis on efficient data utilization and simple model architectures can inspire similar approaches in ASR research to overcome data-related limitations.

Yang, R., Lv, K., Huang, Y., Sun, M., Li, J., & Yang, J. (2023). Respiratory Sound Classification by Applying Deep Neural Network with a Blocking Variable. Applied Sciences, 13(6956). https://doi.org/10.3390/app13126956

• Summary: The paper introduces a deep neural network named Blnet for classifying respiratory sounds, incorporating features from ResNet, GoogleNet, and self-attention mechanisms to tackle the non-IID (not independently and identically distributed) data problem and imbalanced data issues. The model demonstrated improved performance on the ICBHI 2017 respiratory sound database, showcasing a significant advancement in sensitivity and specificity rates over existing methods.
• RQ: How can a deep neural network be optimized for classifying respiratory sounds to facilitate the early detection of respiratory diseases, considering challenges such as non-IID data and imbalanced datasets?
• Hypothesis: The integration of ResNet, GoogleNet, and self-attention mechanisms into a deep neural network, alongside a two-stage training process and mix-up data augmentation within clusters, can significantly improve the classification accuracy of respiratory sounds, even with imbalanced and non-IID data challenges.
• Conclusion: The Blnet model successfully addressed the challenges of non-IID and imbalanced datasets in respiratory sound classification, achieving a 4.22% improvement in average score and a 12.61% improvement in sensitivity over state-of-the-art results. This performance enhancement underscores the efficacy of the proposed network architecture and training strategies.
• Critical observations:
  • The two-stage training process and the introduction of a blocking variable proved effective in managing non-IID data, suggesting the importance of considering data distribution in deep learning models.
  • Mix-up data augmentation within clusters and the use of multiple input transformations (STFT and WT) were critical in addressing data imbalance and enhancing model robustness.
  • The self-attention mechanism played a key role in capturing global dependencies within the data, improving the model's feature extraction capabilities.
  • Simplifying the loss function to handle a four-class classification task as two independent binary classification tasks was found to enhance training effectiveness.
• Relevance: The techniques and findings of this study have direct implications for ASR systems, particularly in enhancing model performance with non-IID and imbalanced datasets. The methods for improving feature extraction and classification in the context of respiratory sound analysis can inform approaches to noise reduction, signal processing, and robust model training in ASR technologies. Furthermore, the attention mechanisms and data augmentation strategies could be adapted to improve ASR systems' ability to deal with diverse and challenging acoustic environments.

Zhou, Rui, Xian Li, Ying Fang, and Xiaofei Li. “Mel-FullSubNet: Mel-Spectrogram Enhancement for Improving Both Speech Quality and ASR.” arXiv, February 21, 2024. http://arxiv.org/abs/2402.13511.

• Summary: This paper introduces Mel-FullSubNet, a network designed for enhancing speech quality and automatic speech recognition (ASR) performance. It focuses on improving both the clarity of speech and its recognizability by machines in noisy conditions. The technique enhances Mel-spectrograms of speech, which can then be used directly for ASR or converted back into speech waveforms using a neural vocoder. The method combines full-band and sub-band network processing, proving to be more effective for ASR and speech quality enhancement compared to previous approaches.
• RQ: Can Mel-spectrogram enhancement via Mel-FullSubNet improve both speech quality and automatic speech recognition performance in noisy conditions?
• Hypothesis: By enhancing Mel-spectrograms using the Mel-FullSubNet, which combines full-band and sub-band processing, both speech quality and ASR performance can be significantly improved in noisy environments.
• Conclusion: Mel-FullSubNet successfully enhances speech quality and ASR performance, outperforming several existing methods. It shows particular strength in providing cleaner speech signals and more accurate ASR results by focusing on Mel-spectrogram enhancement and efficiently leveraging neural vocoders for waveform generation.
• Critical observations:
  • Mel-FullSubNet demonstrates superior generalization to unseen data and environments, a critical advantage for real-world applications.
  • The method's efficacy is underscored by its performance on various datasets, indicating its robustness and adaptability.
  • While Mel-FullSubNet requires more computational resources due to its neural vocoder component, its efficiency and output quality justify the additional cost.
• Relevance:This study is directly relevant to the topic to the challenge of enhancing speech recognition systems in noisy conditions, a common problem in real-world applications. By focusing on Mel-spectrogram enhancement, Mel-FullSubNet provides a novel approach that benefits both speech clarity and ASR accuracy, making it a valuable reference for further research in speech processing technology.

Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., & Poria, S. (2019). Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). arXiv:1906.01815v1.

• Summary: The paper introduces a novel approach to sarcasm detection by leveraging multimodal data. Recognizing that sarcasm often involves incongruities not just in text but also in vocal tone and facial expressions, the authors propose the first dataset, MUStARD, for sarcasm detection using audio-visual cues alongside textual data. This dataset, compiled from popular TV shows, is annotated for sarcasm, aiming to facilitate the development of models that can better understand sarcasm through the integration of multiple modes of communication.
• RQ: How can incorporating multimodal cues (textual, audio, and visual) improve the automatic classification of sarcasm compared to relying on textual data alone?
• Hypothesis: The paper hypothesizes that the inclusion of multimodal information (audio and visual cues, along with textual data) can significantly enhance the performance of sarcasm detection models, reducing the relative error rate by up to 12.9% in F-score when compared to models that use only individual modalities.
• Conclusion: The research demonstrates that multimodal models significantly outperform unimodal variants in sarcasm detection, with a notable reduction in error rate. The findings underscore the importance of considering multiple communication cues, beyond just text, for effectively identifying sarcasm. The MUStARD dataset is also introduced as a valuable resource for future research in multimodal sarcasm detection.
• Critical Observations:

1. Sarcasm detection benefits from multimodal analysis, including textual, audio, and visual data, highlighting the complex nature of sarcasm as a communicative act that often relies on the interplay of various signals.
2. The MUStARD dataset fills a critical gap in research resources, providing a foundation for exploring how different modalities contribute to the understanding of sarcasm.
3. The study's methodology, focusing on a balanced dataset and robust multimodal feature extraction techniques, sets a precedent for future work in this area.

• Relevance: This research is highly relevant to my thesis topic. It pushes the boundaries of sarcasm detection by moving beyond text analysis to include audio and visual cues, offering insights into more holistic approaches to understanding human communication. The findings and the MUStARD dataset can significantly impact the development of more nuanced and effective computational models for detecting sarcasm and other complex emotional or figurative language use cases.

Zhang, Yazhou, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. “CMMA: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations,” n.d.

• Summary: This study introduces the CMMA dataset for benchmarking multi-affection detection in Chinese multi-modal conversations, focusing on sentiment, emotion, sarcasm, and humor. The dataset comprises annotations from a variety of TV series to reflect diverse affective expressions and supports both single-task and multi-task learning paradigms for affective computing research.
• RQ: How multi-modal cues and conversational context influence the detection of multiple affects, including sentiment, emotion, sarcasm, and humor, in Chinese multi-party conversations?
• Hypothesis: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations" likely centers on the premise that incorporating multi-modal data (text, video, audio) and conversational context significantly improves the accuracy and effectiveness of detecting multiple affects (sentiment, emotion, sarcasm, humor) in multi-party conversations. The study posits that the interplay between different modalities and the contextual understanding of conversations enhances the model's ability to interpret complex human affective expressions.
• Conclusion: The findings demonstrate that conversational context and multi-modal data significantly enhance affect detection tasks. The study also highlights the importance of multi-affect annotation for understanding complex human communications, suggesting the CMMA dataset as a valuable resource for future affective computing research.
• Critical observations: While the dataset offers comprehensive insights into multi-affect detection, its focus on Chinese TV series may limit its applicability across different linguistic and cultural contexts. Additionally, the inherent subjectivity of affect annotation poses challenges to achieving unbiased affect detection.
• Relevance: This study is pertinent to my thesis as it provides an opportunity to delve into how various feature fusion methods impact the accuracy of sarcasm recognition in Mandarin using multimodal data. Additionally, the CMMA dataset is highly beneficial to my research because it is among the few Chinese datasets that include labels for sarcasm, offering a valuable resource for studying sarcasm recognition within Mandarin-specific contexts using multimodal information.

Patel, T., & Scharenborg, O. (2024). Improving End-to-End Models for Children’s Speech Recognition. Applied Sciences, 14(6), 2353.

• Summary: Children’s Speech Recognition (CSR) is challenging due to variable speech patterns and limited annotated data. We aim to enhance CSR when no child speech data is available. Traditionally, Vocal Tract Length Normalization (VTLN) mitigates acoustic mismatch in hybrid systems, while End-to-End (E2E) systems use data augmentation. We investigate speed perturbations, spectral augmentation, and VTLN in E2E CSR systems across Dutch, German, and Mandarin. Our experiments show that speed perturbations and spectral augmentation significantly improve performance, with VTLN offering further enhancements while maintaining adult speech recognition. VTLN benefits both genders and is particularly effective for younger children.
• RQ: How to enhance SCR performance while maintaining performance on adults’ speech when adapting the model to children’s speech?
• Hypothesis: VLTN, speed perturbation, and spectral augmentation can be useful.
• Conclusion: VLTN is used for the 1st times to improve E2E CSR work augmentation and normalization enhance CSR task performance the performance of adult speech is largely preserved similar observations in all 3 languages
• Critical observations: Because VTLN needs to be trained independently and then used as a processing step after feature extraction to warp the features for training the ASR network architecture, it may not be compatible with architectures that utilize raw waveform data rather than features. As a result, integrating VTLN into such architectures requires further exploration.
• Relevance: The study's focus on improving Automatic Speech Recognition (ASR) for children's speech, despite limited annotated data, holds relevance to the endeavor of enhancing ASR performance for older adults. Both populations present challenges due to variability in speech patterns and the scarcity of annotated data. Techniques explored in the study, such as Vocal Tract Length Normalization (VTLN) and data augmentation, offer potential solutions that could be adapted to address age-related changes in older adults' speech. Comparative analyses across languages and considerations of age and gender factors provide valuable insights applicable to developing tailored ASR systems for the older adult population. Overall, the study's methodologies and findings offer valuable parallels and considerations for researchers aiming to improve ASR performance for older adults.

Geng, M., Xie, X., Liu, S., Yu, J., Hu, S., Liu, X., & Meng, H. (2022). Investigation of data augmentation techniques for disordered speech recognition. arXiv preprint arXiv:2201.05562.

• Summary: The final speaker adapted system constructed using the UASpeech corpus and the best augmentation approach based on speed perturbation produced up to 2.92% absolute (9.3% relative) word error rate (WER) reduction over the baseline system without data augmentation, and gave an overall WER of 26.37% on the test set containing 16 dysarthric speakers.
• RQ: systematically investigate different data augmentation techniques for disordered speech recognition.
• Conclusion: It suggests that speed-perturbation based augmentation produces the largest improvement in system performance despite the huge mismatch between normal and disordered speech.
• Critical observations:  They increased the amount of speed perturbation data to four times and six times, with only dysarthric speech being processed, the mean WER showed that four times the amount of the original data made the model performance better than six times (4x: 29.47, 6x: 29.52). More augmented data cannot further improve the model performance. In addition, increasing the augmented data from two to four times only reduced the WER by 0.2%. They did not further increase the amount of augmented data, while according to the results when only dysarthric speech data was augmented, it is doubtful whether more data can still lower the WER. This can be explored in future studies by increasing the amount of augmented data from one to six or more times while keeping all other factors the same.
• Relevance: The study exploring data augmentation techniques for dysarthric speech recognition offers insights applicable to improving ASR performance for older adults. By addressing challenges common to both dysarthric speech and speech from older adults, such as variations in speech patterns and articulation, the study provides valuable methodologies and findings. Specifically, the effectiveness of techniques like speed perturbation-based augmentation in enhancing ASR performance underscores their potential utility in optimizing systems for recognizing older adult speech. Furthermore, the study's identification of augmentation limitations and suggestions for future research pave the way for continued refinement of ASR systems tailored to the unique characteristics of older adult speech.

Synthesis

The articles reviewed collectively contribute to the ASR field, showing a trend towards multimodal data use, context awareness, and noise reduction techniques to address complexities in human speech such as sarcasm and humor. Key observations include the importance of integrating audio, visual, and textual data for better sarcasm detection, the effectiveness of dual-channel noise reduction in vehicular environments, the application of deep learning for respiratory sound classification and speech enhancement in noisy settings, and data augmentation techniques in improving ASR performances for a specific group of speakers. Challenges mentioned across these studies involve data scarcity, handling diverse dialects, and computational demands. Future research directions suggest a focus on improving ASR systems' adaptability across languages, cultures and groups, better managing non-IID and imbalanced data, and enhancing emotional intelligence in speech recognition. These findings indicate ongoing efforts to make ASR technologies more intuitive and effective in complex human-machine interactions.

Contributors

Contributors: A list of contributors by contribution

• Article Wang et al. (2023): Yaling Deng
• Article Sungjoo Ahn and Hanseok Ko (2005): Dongwen Zhu
• Article Zhang and Qian (2023): Dongwen Zhu
• Article Zhou et al. (2024): Dongwen Zhu
• Article Bae et al. (2023): Soogyeong Shin
• Article Gairola et al. (2021): Soogyeong Shin
• Article Yang et al. (2023): Soogyeong Shin
• Article Castro et al. (2019) : Erin Shi
• Article Zhang et al. (2021): Youyang Cai
• Article Patel, T., and Scharenborg, O. (2024): Wansu Zhu
• Article Geng et al. (2022): Wansu Zhu
• Introduction: All
• Synthesis: All

ASR II

Introduction

In the realm of automatic speech recognition, two distinct, yet in a way connected topics have attracted limited attention: whispering speech and child speech recognition. Both whispering speech and child voices exhibit unique acoustic characteristics that differ from typical, neutral speech, and thus require special attention and approaches. Whispering speech poses unique challenges due to its reduced dynamic range and spectral variations, while children's speech often lacks the articulation found in adult speech, which can complicate the task of separating their voices in noisy environments. Recent advancements we discuss below have proven useful in these two domains, which underscores the ongoing efforts to improve the accuracy, robustness, and general adaptability of ASR and speech technology in general in diverse linguistic and environmental contexts.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Park, D. S., Chan, W., Zhang, Y., Chiu, C. C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019). Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779.

• Summary: The paper introduces SpecAugment, a straightforward data augmentation method for speech recognition tasks that operates directly on the feature inputs of a neural network. The method consists of time warping, frequency masking, and time masking applied to the log-mel spectrogram. This approach, despite its simplicity, achieves state-of-the-art results on the LibriSpeech 960h and Switchboard 300h datasets, outperforming more complex systems even without the use of Language Models.
• RQ: Can simple, computationally easy data augmentation techniques applied directly to the feature inputs of a neural network improve the performance of end-to-end automatic speech recognition systems?
• Hypothesis: Applying augmentation techniques such as time warping or time/frequency masking,directly on the log mel spectrogram may enhance the robustness and performance of speech recognition models, making them less prone to overfitting and more generalizable to various speech inputs.
• Conclusion: SpecAugment substantially enhances the performance of ASR systems, achieving top results on major speech recognition benchmarks even without the necessity for external language models, achieving 6.8% Word Error Rate, beating the previous results of state-of-the-art solutions with 7.5% WER.
• Critical observations: The least impactful contribution of time warping (compared to frequency/time masking) implies that, under constraints, time warping could be omitted. However, it still may be practical for whispering speech recognition where the temporal dynamics might differ from normal speech.
• Relevance: For whispering speech recognition, SpecAugment's ability to improve model generalization and robustness with minimal data could be particularly useful, addressing the common issue of data scarcity in this domain and making the model more robust to variations within whispered speech. Additionally, the simplicity of implementing SpecAugment allows easy integration into existing speech recognition frameworks such as Whisper model.

Wang, C., Wu, Y., Du, Y., Li, J., Liu, S., Lu, L., ... & Zhou, M. (2020). Semantic mask for transformer based end-to-end speech recognition. arXiv preprint arXiv:1912.03010.

• Summary: The article presents a semantic mask-based augmentation approach for improving end-to-end ASR systems. This method involves masking the input features corresponding to specific output tokens, such as words or word-pieces, during training (similar to how BERT is trained with its [MASK] token). The objective is to force the model to predict the masked tokens using contextual information, with this enhancing the model's generalization capabilities. Experiments on the Librispeech 960h and TedLium2 datasets demonstrated state-of-the-art performance, showing the effectiveness of this approach.
• RQ: Can the generalization capacity and language modeling power of end-to-end ASR models be improved with the employment of an NLP technique of semantic masking?
• Hypothesis: By applying a semantic mask to mask out input features corresponding to specific output tokens, the models will be encouraged to rely more on contextual information, improving their modeling capabilities and generalization.
• Conclusion: The introduction of a semantic mask in transformer-based E2E ASR models leads to significant improvements in WER on the Librispeech and TedLium2 datasets. This approach enhances the model's ability to use contextual information and strenghtens its robustness to various acoustic distortions, which potentially can be useful for the task of whispering speech recognition as well.
• Critical observations: The semantic mask approach is particularly effective in challenging conditions, where reliance on contextual information becomes crucial for accurate token prediction, so I may assume it could prove useful in whispering speech too, where one word could be more prominent than the other. However, while the paper describes the semantic masking strategy, further details on how tokens could be selected for masking and the criteria for that could enhance reproducibility and allow for more detaield analysis of why this strategy works.
• Relevance: Semantic Masking's emphasis on enhancing a model's reliance on contextual information rather than solely on acoustic features suggests that it could be relevant for whispering speech recognition. Whispered speech which is characterized by reduced dynamic range and spectral variations, presnts unique challenges that, I guess, might be mitigated by a model better attuned to contextual cues, where one part of the utterance might be more prominent than the other.

Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J.(2021) ATTENTION IS ALL YOU NEED IN SPEECH SEPARATION. arXiv:2010.13154

• Summary: This article introduces SepFormer, a Transformer-based architecture for speech separation that does not rely on Recurrent Neural Networks (RNNs). By employing a multi-scale approach with transformers to learn both short and long-term dependencies, SepFormer sets new state-of-the-art performance on WSJ0-2mix and WSJ0-3mix datasets. It demonstrates an SI-SNRi of 22.3 dB on WSJ0-2mix and 19.5 dB on WSJ0-3mix, benefiting from the parallelization capabilities of Transformers, which allow for faster processing and reduced memory demands compared to RNN-based models.

• RQ: Can a Transformer-based architecture, without RNNs and employing a multi-scale approach, achieve state-of-the-art performance in speech separation tasks?

• Hypothesis: The authors hypothesize that SepFormer, by leveraging a dual-path framework with transformers to model both short and long-term dependencies, can outperform existing RNN-based speech separation models in both effectiveness and efficiency.

• Conclusion: The SepFormer architecture achieves state-of-the-art performance on standard speech separation datasets, confirming the hypothesis that Transformers can efficiently model temporal dependencies for speech separation tasks. It also demonstrates a significant advantage in terms of processing speed and memory usage due to its parallelizable nature and effectiveness even with downsampling.

• Critical observations: The success of SepFormer underscores the limitation of RNNs in handling long sequences and their inability to parallelize computations effectively. It highlights the importance of modeling both short and long-term dependencies in speech separation tasks, with the dual-path framework providing a robust solution. However, he datasets used (WSJ0-2mix and WSJ0-3mix) are standard benchmarks but may not fully represent all real-world scenarios or challenges in speech separation tasks, such as varied noise conditions, different numbers of speakers, or non-ideal recording environments.

• Relevance: This research contributes significantly to the fields of speech processing and automatic speech recognition by demonstrating the effectiveness of Transformer-based models in speech separation tasks. It paves the way for future exploration of non-RNN architectures in audio processing and opens up new possibilities for real-time speech separation applications, benefiting a wide range of technologies from voice-activated assistants to hearing aids.

Kuan-Hsun Ho, Jeih-weih Hung, Berlin Chen(2023). ConSep: a Noise- and Reverberation-Robust Speech Separation Framework by Magnitude Conditioning. arXiv:2403.01792.

• Summary: This research introduces ConSep, an innovative framework designed to enhance speech separation capabilities in challenging acoustic environments characterized by noise and reverberation. Unlike traditional methods that primarily focus on time-domain techniques, ConSep uniquely integrates magnitude conditioning with a dual-encoder approach, effectively leveraging the strengths of both time and frequency domain features. The framework is rigorously evaluated across various conditions, including anechoic, noisy, and reverberant settings, demonstrating superior performance compared to existing models such as SepFormer and Bi-Sep.
• RQ: Can a speech separation model designed with a magnitude-conditioned time-domain framework and dual-encoder strategy, achieve superior performance in noisy and reverberant environments compared to Sepformer?
• Hypothesis: The study hypothesizes that the integration of magnitude conditioning with a dual-encoder approach, which leverages both time and frequency domain features, will significantly improve speech separation performance, especially in challenging acoustic settings.
• Conclusion: ConSep outperforms established models by a significant margin across multiple testing environments, including anechoic, noisy, and reverberant conditions. The framework's innovative approach to leveraging magnitude spectrograms for conditioning, combined with the dual-encoder system, effectively addresses the limitations of previous models.
• Critical observations: The effectiveness of ConSep is particularly notable in environments where noise and reverberation traditionally complicate speech separation tasks, highlighting the importance of combining features from both the time and frequency domains to capture a more comprehensive set of characteristics for accurate speech separation.While ConSep shows remarkable performance improvements, the study also suggests areas for further refinement, such as optimizing computational efficiency for real-time applications and exploring the model's adaptability to a wider range of acoustic scenarios.
• Relevance: This research holds significant relevance for the fields of ASR and speech processing, particularly in developing robust systems capable of operating in acoustically adverse environments. ConSep's methodology provides a promising direction for future innovations in speech separation technology, with potential applications in voice-activated systems and assistive technologies for individuals with hearing impairments.

HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

• Summary: The HiCMAE framework pioneers a self-supervised approach for Audio-Visual Emotion Recognition (AVER), leveraging unlabeled data through hierarchical learning, masked modeling, and contrastive learning. Surpassing traditional methods, HiCMAE sets new benchmarks in AVER by addressing data scarcity and improving representation quality, demonstrating the significant potential of self-supervised learning in speech and emotion recognition.
• RQ: Can a self-supervised learning model, specifically designed with hierarchical contrastive masked autoencoding, effectively utilize unlabeled audio-visual data to significantly advance the field of AVER?
• Hypothesis: The HiCMAE framework demonstrates a significant improvement over existing state-of-the-art methods in AVER. Through extensive experimentation across multiple datasets, it is established that HiCMAE not only achieves better performance in both categorical and dimensional AVER tasks but also highlights the efficacy and potential of self-supervised learning strategies in speech technology.
• Conclusion: The HiCMAE framework demonstrates a significant improvement over existing state-of-the-art methods in AVER. Through extensive experimentation across multiple datasets, it is established that HiCMAE not only achieves better performance in both categorical and dimensional AVER tasks but also highlights the efficacy and potential of self-supervised learning strategies in speech technology.
• Critical observations: HiCMAE's unique hierarchical approach, incorporating skip connections and cross-modal contrastive learning, addresses critical challenges in learning representations from unlabeled data. The framework significantly outperforms traditional supervised and self-supervised methods, underlining the advantages of its novel methodology. Despite its strengths, the performance of HiCMAE heavily relies on the diversity and quality of the pre-training datasets, suggesting areas for future improvement and exploration.
• Relevance: The advancements demonstrated by the HiCMAE framework are not merely confined to AVER but extend broadly to the field of speech technology. By showcasing the potential of self-supervised learning in overcoming data scarcity and enhancing emotion recognition, HiCMAE sets a precedent for future research and development in creating more emotionally aware and interactive speech-based systems.

MMER: Multimodal Multi-task Learning for Speech Emotion Recognition

• Summary: MMER introduces a novel framework in Speech Emotion Recognition (SER), combining multimodal inputs (speech and text) and multi-task learning to achieve state-of-the-art performance. It incorporates auxiliary tasks—Automatic Speech Recognition (ASR), Supervised Contrastive Learning (SCL), and Augmented Contrastive Learning (ACL)—to enrich the model's understanding and recognition of emotions.
• RQ: How can the integration of multimodal inputs and multi-task learning strategies improve the performance of speech emotion recognition systems?
• Hypothesis: The combination of textual and acoustic information, alongside auxiliary learning tasks, will significantly enhance SER by providing a more comprehensive dataset for emotion recognition.
• Conclusion: MMER introduces a novel approach to Speech Emotion Recognition (SER), significantly outperforming existing models on the IEMOCAP benchmark. It combines multimodal data integration and multi-task learning, demonstrating the effectiveness of leveraging both speech and text data, alongside auxiliary tasks, for enhanced emotion recognition. This strategy effectively addresses the prosodic bias in speech, presenting a substantial advancement in SER. However, MMER's reliance on large batch sizes for training and pre-computed text features poses challenges, including computational resource demands and limitations in real-time applicability. Future efforts will focus on mitigating these constraints, aiming to refine and expand MMER's capabilities for broader and more efficient use in SER applications.
• Critical observations:The MMER model outperforms existing SER approaches by effectively leveraging both speech and text data. This multimodal strategy addresses speech's prosodic bias, offering a richer feature set for accurate emotion detection. The auxiliary tasks, particularly SCL and ACL, refine the model's capacity to capture emotion-specific and speaker-invariant features, showcasing the value of multi-task learning in deepening emotion understanding. Despite its advantages, MMER's complexity poses challenges in model interpretability and computational efficiency.
• Relevance: MMER's advancements underscore the importance of emotional intelligence in human-computer interaction, demonstrating how multimodal data and multi-task learning can elevate SER systems. This approach aligns with the imperative for computers to understand and respond to human emotions, suggesting a promising direction for future SER research and the development of empathetic HCI technologies.

ShEMO: A Large-Scale Validated Database for Persian Speech Emotion Detection

• Summary: ShEMO introduces a validated, semi-natural Persian speech database, drawing from online radio plays. It encompasses 3 hours and 25 minutes of audio across 3000 utterances from 87 speakers, covering six emotions. Validation involved a majority vote among twelve annotators, achieving a 64% inter-annotator agreement.
• RQ: A diverse and accurately annotated speech database will significantly improve speech emotion recognition in Persian.
• Hypothesis: The combination of textual and acoustic information, alongside auxiliary learning tasks, will significantly enhance SER by providing a more comprehensive dataset for emotion recognition.
• Conclusion: The ShEMO database significantly enriches Persian speech emotion research by providing a comprehensive collection of semi-natural emotional and neutral speech samples. It sets a new benchmark for future studies with its validated dataset and baseline results from standard classification methods. Looking ahead, efforts will focus on broadening the database with more fear utterances, employing advanced classification techniques like deep neural networks, and enriching annotations with dimensions of arousal, valence, and emotional intensity. This groundwork is expected to catalyze further innovation in speech emotion detection, enhancing the understanding and development of more responsive and emotionally aware systems.
• Critical observations:ShEMO's semi-natural origin offers a realistic dataset for emotion recognition systems. The substantial annotation process ensures data reliability, a prerequisite for training precise models. However, the dataset's emotion imbalance and the exclusion of underrepresented emotions, like fear, might skew model biases. The challenge of fully capturing natural speech emotions remains.
• Relevance: ShEMO enriches speech technology by addressing Persian emotional speech's under-researched area. It underpins the need for language-specific databases in accurately interpreting speech and emotion, thereby facilitating more nuanced human-computer interactions.

Synthesis

In conclusion, all these studies underscore the importance of innovative approaches in enhancing ASR systems' performance and robustness, finding the necessary tricks to solve the complexities dictated by challenging acoustic features and environments. They demonstrate the potential of data augmentation and speech separation, and push the boundaries of what's achievable in speech recognition tasks in general through focusing on very specific tasks.

Through these works, a noticeable shift from conventional RNN-based structures to Transformer models can be noticed, as evidenced by SepFormer and ConSep. These models take advantage of the ability to process sequences in parallel, resulting in significant improvements in efficiency and scalability. The use of techniques such as SpecAugment and semantic masks, in turn, highlights the increasing recognition of the importance of robust data augmentation in conditions of insufficient data. These methods improve model generalisation, enabling systems to handle a wider variety of speech inputs more effectively.

There is an ongoing debate about the relative contribution of different augmentation techniques, such as the impact of time warping versus time masking. This debate highlights the need for a better understanding of how different aspects of speech data contribute to model learning and performance. The integration of external language models with ASR systems is a also topic of separate discussion: although research has shown remarkable performance without them, the debate continues on the best approach to find and keep contextual information for speech recognition. When it comes to child speech recognition, a debate might arise around the scalability of ConSep vs SepFormer to handle larger datasets, questioning whether ConSep's specialized approach or SepFormer's more generalized framework is better suited for future advancements in ASR technology.

We think that in the future, research should focus on integrating multi-modal data and enhancing adaptation to diverse acoustic environments, and the studies reviewed are certainly a step towards at least the latter. We are sure that the combination of audio and visual data would present new opportunities for improving speech recognition in such challenging settings: for whispering speech recognition, for instance, there already exists a database called Audiovisual Whisper which audios and videos of whispering, and much work is being done in that direction. However, even though there's a lot work ahead, this short list of studies that we discussed here already shows big steps forward in speech technology, opening doors to more flexible speech recognition systems that are better suited for everyday use.

Contributors

Contributors: A list of contributors by contribution

• Article Park et al. (2019): Igor Marchenko
• Article Wang et al. (2020): Igor Marchenko
• Article Subakan et al. (2021): Wenjun Meng
• Article Kuan-Hsun et al. (2023): Wenjun Meng
• Article Nezami et al. (2019): Jingwen Shi
• Article Ghosh et al. (2023): Jingwen Shi
• Article Sun et al. (2024): Jingwen Shi

• Introduction: Igor Marchenko & Wenjun Meng
• Synthesis: Wenjun Meng & Igor Marchenko

Speech Enhancement

Introduction

Speech enhancement/restoration represent pivotal areas within the field of speech technology, focusing on the improvement and rehabilitation of speech signals that have been degraded by various factors such as noise, reverberation, and data compression. The significance of this thematic focus cannot be overstated, as it directly impacts the usability, intelligibility, and overall quality of speech communication in diverse contexts, including telecommunications, voice assistants, and hearing aids. This literature collection aims to compile the most recent and influential works that drive innovation in these domains, highlighting the cutting-edge methodologies and the transformative potential they hold for enriching human-computer interaction and ensuring the accessibility of speech-based services for all users.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Donahue, C., Li, B., & Prabhavalkar, R. (2018). Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (arXiv:1711.05747). arXiv. http://arxiv.org/abs/1711.05747

• Summary: This paper investigates the application of Generative Adversarial Networks (GANs) for speech enhancement, particularly for improving the noise robustness of ASR systems. Through comprehensive experiments, it introduces a frequency-domain approach (FSEGAN) to speech enhancement that shows improved ASR performance over traditional time-domain methods (SEGAN).
• RQ: Can GAN-based speech enhancement techniques effectively improve the noise robustness of ASR systems compared to traditional noise suppression methods?
• Hypothesis: The paper hypothesizes that GAN-based speech enhancement, especially when operating on log-Mel filterbank spectra rather than waveforms, will provide significant improvements in ASR system performance in noisy conditions.
• Conclusion: The study concludes that while GAN-based speech enhancement methods, particularly FSEGAN, can improve ASR performance in noisy conditions, they do not outperform multi-style training (MTR) methods. Retraining the ASR system using both the original noisy audio and the audio improved by GANs leads to better performance. This suggests that GAN-enhanced audio could be a valuable addition to improve ASR systems when used alongside the original noisy input.
• Critical observations: SEGAN, while effective in removing additive noise, is less effective in reverberant conditions compared to the frequency-domain approach (FSEGAN). On the contrast, FSEGAN significantly improves ASR performance but does not outperform traditional MTR alone. However, combining noisy and enhanced features for retraining enhances the system's robustness.
• Relevance: This article is relevant to techniques used to bolster the performance of ASR systems, highlighting the significant potential of innovating GAN-based model in this field.

Y. Koizumi, H. Zen, S. Karita, et al. (2023). Miipher: A robust speech restoration model integrating self-supervised speech and text representations, arXiv:2303.01664.

• Summary: The paper presents Miipher, a robust speech restoration (SR) model that integrates self-supervised speech and text representations to enhance the quality of degraded speech signals. The model is designed to address two primary challenges in SR: phoneme masking and deletion.
• RQ: How to develop a robust speech restoration (SR) model that can convert degraded speech signals into high-quality ones, with a focus on handling difficult degradations such as phoneme masking and deletion?
• Hypothesis: The proposed SR model, Miipher, will be robust against various audio degradations and enable the training of high-quality text-to-speech (TTS) models from restored speech samples.
• Conclusion: The study concludes that Miipher is effective in restoring speech samples in-the-wild and can increase the value of speech samples by improving their quality as training data for speech generation tasks.
• Critical observations: The use of w2v-BERT features significantly improves SR performance compared to log-mel spectrogram-based methods, the effectiveness of PnG-BERT features in preserving text content, and the importance of speaker embeddings for retaining speaker characteristics in restored speech.
• Relevance: The relevance of this study is significant for the field of speech enhancement/restoration, as it demonstrates a method to enhance the quality of existing speech datasets and expand the potential applications of non-studio speech recordings.

Vinith Kishore, Nitya Tiwari, and Periyasamy Paramasivam. “Improved Speech Enhancement Using TCN with Multiple Encoder-Decoder Layers”. In: Interspeech 2020. ISCA. 2020, pp. 4531–4535. doi: 10.21437/Interspeech.2020-3122. url: https://doi.org/10.21437/Interspeech.2020-3122.

• Abstract: This paper presents a deep learning-based single-channel speech enhancement technique that utilizes a multilayer encoder-decoder structure and a Temporal Convolutional Network (TCN) to improve the quality of speech for applications such as smart speakers and voice assistants. The technique leverages the encoder-decoder to obtain a representation suitable for speech enhancement and employs a TCN-based separator between the encoder and decoder to learn long-range dependencies. The optimal number of encoder-decoder layers is determined through t-SNE analysis of the representations learned by different architectures. Experimental results show that the proposed two-layer encoder-decoder structure achieved a 48% improvement in Word Error Rate (WER) over unprocessed noisy data and improvements of 33% and 44% in WER over two baseline models.
• Research Question (RQ): The research question focuses on exploring the effectiveness of the multilayer encoder-decoder structure in the task of single-channel speech enhancement and the role of TCN in learning long-range dependencies for separating noise and clean speech. Additionally, the study aims to determine the optimal number of encoder-decoder layers for effective noise suppression and speech enhancement.
• Hypothesis: The paper hypothesizes that using a multilayer encoder-decoder structure can obtain a noise-independent representation, which is useful for separating clean speech and noise. It is also hypothesized that TCN can effectively learn long-range dependencies in the encoded output and provide an enhanced speech mask, thereby improving the performance of speech enhancement.
• Conclusion: The conclusion indicates that the proposed two-layer encoder-decoder structure outperforms unprocessed noisy data and two baseline models in objective measures of speech quality (such as PESQ and SI-SNR) and Word Error Rate (WER) on a speech recognition platform. Furthermore, t-SNE analysis demonstrates that the two-layer structure can learn a representation suitable for speech enhancement applications.
• Critical Observation: Although the proposed architecture has achieved significant improvements in speech enhancement, the study mainly focuses on specific types of noise and speech datasets, which may not fully represent the diverse noise conditions in the real world. Moreover, increasing the number of encoder-decoder layers could lead to an increase in the number of model parameters, thereby increasing computational costs and the risk of overfitting. Future work needs to explore model optimization and compression techniques to reduce the number of parameters and test the generalizability and suitability of the technique in unseen noisy environments.

• Relevance : The research is closely related to the field of speech enhancement, especially in improving the performance of Automatic Speech Recognition (ASR) systems in noisy environments. By processing signals directly in the time domain using deep learning techniques, the study provides a new perspective and approach for designing effective single-channel speech enhancement systems. Additionally, by comparing the performance of different architectures, this paper offers guidance for selecting the appropriate model structure and number of layers, which is significant for developing efficient and accurate speech enhancement algorithms.

Asiedu Asante, B. K., Broni-Bediako, C., & Imamura, H. (2023). Exploring multi-stage gan with self-attention for speech enhancement. Applied Sciences, 13(16), 9217. https://doi.org/10.3390/app13169217

• Abstract: This paper explores the integration of self-attention mechanisms into multi-stage generative adversarial networks (GANs) for speech enhancement. The authors empirically study the effect of adding self-attention to the convolutional layers of the generators in two existing multi-stage GAN architectures: ISEGAN and DSEGAN. The experimental results demonstrate that incorporating self-attention leads to improvements in speech enhancement quality and intelligibility across objective evaluation metrics. The paper also finds that adding self-attention to ISEGAN's generators improves its performance to be competitive with DSEGAN while using a smaller model size.

• Research Questions:

1. Can integrating self-attention mechanisms into multi-stage speech enhancement GANs improve their enhancement performance?
2. How does the incorporation of self-attention affect the performance gap between ISEGAN and DSEGAN architectures?

• Hypothesis: The authors hypothesize that introducing self-attention into the convolutional layers of the generators in multi-stage speech enhancement GANs will allow the models to better capture temporal dependencies in the input signal sequences, leading to improved enhancement quality. They also posit that adding self-attention to ISEGAN may allow it to approach the performance of the larger DSEGAN model.

• Conclusion: The experimental results confirm that integrating self-attention mechanisms into the ISEGAN and DSEGAN architectures (referred to as ISEGAN-Self-Attention and DSEGAN-Self-Attention) leads to consistent improvements in objective speech enhancement metrics. Furthermore, ISEGAN-Self-Attention is able to achieve enhancement performance competitive with the base DSEGAN model while using only half the model parameters. This highlights the potential of self-attention to improve the efficiency-performance tradeoff in multi-stage speech enhancement GANs.

• Methodology:
  • The paper provides a clear description of how the self-attention mechanism is integrated into the existing ISEGAN and DSEGAN architectures.
  • The experimental setup is reasonable, using a standard dataset (Voice Bank corpus) and evaluation metrics.
  • However, the paper does not include any subjective evaluation (e.g. human listening tests), which would provide additional insight into the perceptual quality of the enhanced speech.

• Results and Argumentation:
  • The objective evaluation results strongly support the paper's conclusions regarding the benefits of integrating self-attention.
  • The authors provide a logical argument for why self-attention is able to improve performance by better capturing temporal dependencies.
  • It would be interesting to see further analysis of how the self-attention mechanisms are operating, e.g. visualizations of the attention weights.

• Potential Biases:
  • The paper only evaluates the proposed approach on a single dataset. Testing on additional datasets would help assess the generalizability of the findings.
  • All experiments use the same hyperparameters for the self-attention mechanisms. It's unclear if these are the optimal settings.

• Relevance: This paper is highly relevant to research on deep learning architectures for speech enhancement, specifically in demonstrating the benefits of integrating self-attention into multi-stage GAN models. The findings regarding the efficiency-performance tradeoff between ISEGAN-Self-Attention and DSEGAN are notable and could inform model selection in practical applications.

Synthesis

The four papers reviewed are dedicated to advancing the field of speech enhancement and restoration, aiming to improve the robustness and performance of speech recognition systems in noisy and degraded environments. The study by Donahue et al. explores the application of Generative Adversarial Networks (GANs) in speech enhancement, particularly their potential to improve the noise robustness of ASR systems. By operating GANs on log-Mel filterbank spectra, the study demonstrates the potential of GANs in improving ASR performance, although it does not surpass traditional multi-style training methods. This work emphasizes the importance of speech enhancement in the frequency domain and points to the possibility of further improving performance by combining GAN-enhanced audio with retrained ASR systems.

Koizumi et al. propose Miipher, a robust speech restoration model that integrates self-supervised speech and text representations, aimed at addressing the issues of phoneme masking and deletion in speech restoration. Miipher increases the potential use of these samples in speech generation tasks by improving the quality of restored speech samples. The study highlights the importance of using w2v-BERT features and speaker embeddings in retaining textual content and speaker characteristics when dealing with various audio degradations.

The work by Vinith Kishore et al. focuses on improving single-channel speech enhancement techniques using multilayer encoder-decoder structures and Temporal Convolutional Networks (TCNs). By determining the optimal number of encoder-decoder layers through t-SNE analysis, the study shows the effectiveness of the two-layer structure in enhancing speech quality and reducing word error rates. However, the study also points out limitations in diverse noise conditions and future directions, including the application of model optimization and compression techniques.

Asante et al. explore the integration of self-attention mechanisms into multi-stage generative adversarial networks (GANs) for speech enhancement. The authors empirically study the effect of adding self-attention to the convolutional layers of the generators in two existing multi-stage GAN architectures: ISEGAN and DSEGAN. The experimental results demonstrate that incorporating self-attention leads to improvements in speech enhancement quality and intelligibility across objective evaluation metrics. The paper also finds that adding self-attention to ISEGAN's generators improves its performance to be competitive with DSEGAN while using a smaller model size, highlighting the potential of self-attention to improve the efficiency-performance tradeoff in multi-stage speech enhancement GANs.

Overall, these studies collectively emphasize the importance of innovative approaches in the field of speech enhancement and restoration, whether through the use of GANs, self-supervised learning, deep learning techniques, or the integration of self-attention mechanisms. The findings from these studies contribute to the ongoing efforts in improving the robustness and performance of speech recognition systems in challenging environments, with potential applications in various domains such as telecommunications, assistive technologies, and human-computer interaction.

Contributors

• Introduction: Janice Huang

• Article Donahue et al.(2018): Ting Zhang
• Article Nitya Tiwari (2020): Ziyun Zhang
• Article Y. Koizumi et al.(2023): Janice Huang
• Article Asiedu Asante(2023): Qing Li
• Synthesis: Ziyun Zhang, Ting Zhang

Miscellaneous

This last section corresponds to articles that did not fit well inside other themes.

Introduction

Voice technology, transcending the traditional boundaries of speech recognition and synthesis, has emerged as a transformative force in a multitude of sectors, revolutionizing not just how we communicate with machines, but also how sound is manipulated and perceived in our digital world. This segment, aptly titled "None of the Above," delves into the innovative applications of voice technology beyond the realms of text-to-speech (TTS) and automatic speech recognition (ASR). It encompasses a wide array of technologies including voice enhancement, noise reduction, accent modification, and speaker seperation, each playing a pivotal role in refining and enriching the auditory experience. These advancements underscore the versatility and depth of voice technology, pushing the boundaries of what is possible in audio quality, clarity, and customization.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Speech Emotion Recognition

Grimm, M., Kroschel, K., & Narayanan, S. (2007, April). Support vector regression for automatic recognition of spontaneous emotions in speech. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07 (Vol. 4, pp. IV-1085). IEEE.

• Summary: The paper presents methods for estimating emotions expressed spontaneously in speech, using Support Vector Regression (SVR). It evaluates three emotion primitives—valence, activation, and dominance—showing SVR's superiority over Fuzzy Logic and Fuzzy k-Nearest Neighbor classifiers in accuracy and correlation with human assessments.
• RQ: How to estimate emotions under the conditions of (1) nonacted, spontaneous speech and (2) non-categorical, quasicontinuous emotional content.
• Hypothesis: SVR can more accurately estimate emotions in speech compared to traditional classifiers, given its ability to handle continuous emotion primitives and complex non-linear relationships in data.
• Conclusion: SVR outperforms Fuzzy Logic and k-Nearest Neighbor classifiers in estimating emotions from speech, achieving lower classification errors and higher correlations with reference emotions. This underscores SVR's suitability for continuous-valued emotion estimation in spontaneous speech.
• Critical observations: SVR yields the lowest mean classification errors and highest correlation coefficients for emotion estimation. In addition, Feature selection indicates that using 20 features suffices for accurate emotion estimation across different classifiers.
• Relevance: This study advances automatic emotion recognition in speech, crucial for improving human-machine interaction and developing emotionally intelligent systems. Future work will investigate designing a real-time system using the algorithms. The advantage of continuous-valued estimates of the emotional state of a person could be used to build an adaptive emotion tracking system that is capable to adapt to individual personalities and long-term moods.

Z. Huang, M. Dong, Q. Mao, and Y. Zhan, “Speech emotion recognition using cnn,” in Proceedings of the 22nd ACM international conference on Multimedia,pp. 801–804,ACM, 2014.

Summary：The paper introduces a CNN model that processes input data in two stages, using unlabeled samples for candidate feature extraction and then learning discriminative features under semi-supervision.

RQ：The main research question is how to efficiently and automatically extract discriminative sentiment features from speech signals for sentiment recognition, especially in complex scenarios where the speaker and environment change.

Hypothesis：The main research question is how to efficiently and automatically extract discriminative sentiment features from speech signals for sentiment recognition, especially in complex scenarios where the speaker and environment change.

Conclusion：The semi-CNN model can effectively learn emotionally skewed features to achieve consistent and robust performance in speech emotion recognition tasks.

Critical observations：Semi-CNN models benefit from a two-stage feature learning process that initially extracts candidate features without labeling the data. The use of novel objective functions to improve feature saliency, orthogonality, and discrimination helps to enhance the robustness of the model.

Relevance：It is important to facilitate human-computer interaction by improving the accuracy and reliability of speech emotion recognition systems. It contributes to the development of the field of affective computing and may influence the development of more sensitive and adaptive SER systems.

Synthetically improving foreign-accented speech recognition

Introduction

More often than not, speech corpora either contain only native speech, or the non-native subset is significantly underrepresented. At the same time, gender and foreign accent are the most salient factors contributing to changes in the acoustics of speech. However, not only are there numerous possible combinations of L1 and L2s, but the annotation and labelling os recordings to a suitable degree (e.g. age of L2 acquisition, country of origin, L1, L2 proficiency, language of education etc. are all factors that should be reported in order to make the speech resources reliable and usable) are laborious and expensive.

In light of these challenges, methods of synthetical data augmentation have been recently explored in the literature. While creating synthetically-accented data through accent conversion models (ACMs) is a straightforward, inexpensive, and oof-the-shelf approach, it is not without limitations and the degree to which recognition performance is improved through such approaches depends on several factors. The following three articles provide some insight into these approaches and highlight both major advantages and persistent challenges.

Zhao et al. (2018): Accent conversion using phonetic posteriograms

Summary: Accent conversion (AC) means transforming non-native speech to sound as if the speaker had a native accent, or vice-versa. The main challenge faced in traditional methods of voice conversion is decoupling the speaker’s voice quality from their pronunciation (i.e. teasing apart accent information and keeping everything else acoustically unchanged). Additionally, when mapping source spectra from a native speaker into the acoustic space of an L2 speaker, previous attempts focus on acoustic similarity: changing formants- and pitch trajectories, blending spectral envelopes. The alternative used here is, in turn, is phonetic similarity, which maps source to target based on an intermediate phonetic label. The phonetic posteriograms are computed using a DNN-based acoustic model. The distance between these phonetic posterior feature vectors is calculated to find the closest pairs of frames between source (native) and target (L2) speakers. The frame pairs are used to train a GMM. The two baselines used are acoustic similarity matching and dynamic time warping.

Experimental setup: get Kaldi DNN acoustic model, train it on Librispeech data, get native English speech (CMUArctic) and non-native recordings (Hindi, Korean, Arabic), use STRAIGHT for speech decomposition, MFCC extraction, train GMMs (128 components), synthesize speech by reconstructing spectrograms and adding aperiodicity.

RQ: How can accent-related features be successfully decoupled from speaker-related features, to achieve non-native to native voice conversion while preserving speech quality?

Results: Synthesized results were compared to baselines through listening tasks using Mechanical Turk (rating acoustic quality, speaker identity y/n, nativeness of resynthesized speech):

• significantly higher acoustic quality ratings compared to baselines.
• comparable speaker identity scores.
• strong preferrence for posteriogram conversions by native EN speakers as more ‘native-like‘ compared to baselines and original L2 utterances.

Critical observations: This paper addressed the opposite issue, namely converting foreign-accented speech to sound like native one (mainly for educational purposes). This still means you need to figure out which features are related to accent, and which features are related to anything else, but is arguably the easier thing to do, as it requires to drop information instead of successfully adding it. Additionally, the approach is not entirely explainable, because posteriograms are encoder features and it's not always transparent what is learned to be most relevant. Lastly, this approach likely works increasingly worse the fewer speakers there are in a dataset. Even if you accented speech data, one speaker can only have one accent, so in case the number of speakers is small, the model might learn to encode speaker identity instead of accent features.

Relevance: It is important to know that given enough speakers and enough data, accent features can be decoupled from other speech features and dropeed to obtain a higher perceived 'nativeness' of the speech.

Jin et al. (2023): Voice-preserving zero-shot multiple accent conversion

Summary: Separating accent from speaker identity is usually the hardest, because each speaker in the dataset has one single accent. Previous attempts at doing this include:

• use adversarial learning to get a discriminator to wipe out speaker-dependent information from content embeddings.
• quantization of different features in speech to obscure undesired information.

The main problem with conventional approaches to conversion is that they very often require available utterances with the same text in both source and target accent, making their applicability very limited. Alternatively, different approaches require either training or fine-tuning on the input utterances.

The current paper uses a pronunciation encoder, an acoustic encoder, and a HiFiGAN voice decoder. During training, the model minimises reconstruction loss between input and output mel-spectrograms. The pronunciation encoder synthesizes accent-dependent pronunciation sequences using accent IDs. The acoustic encoder mapss MFCCs and periodicity features to a single vector, while adversarial training removes accent information. Lastly, the decoder reconstructs waveforms from the processed features. The model is evaluated on audio quality, speaker similarity, and accent conversion effectiveness.

Results: Results indicate it maintains comparable audio quality to the original, maintains speaker similarity, and is efficient in replicating perceived nativeneess. However, listeners struggled to identify synthesized accents if they were unfamiliar with the target language (e.g. a native US listener could not classify a Korean accent on English as such, but a bilingual Korean-American listener could). Overall, the paper presents one of the best performing ACMs, that is able to preserve both speaker identity and acoustic quality during conversion.

Critical observations: I think this paper achives a lot given that it's zero shot, but I am a bit critical about just how 'zero-shot' it truly is. They use a pre-trained acoustic model and while they do not require accent labels or speaker IDs, it seems that their training set contains over 24h of accented speech for all accents that they're synthesizing in. Additionally, none of their code is openly available, which is understandable for a private corporation like Meta, but it's still a bit disappointing.

Klumpp et al. (2023): Synthetic cross-accent data augmentation for ASR

Summary: Foreign-accentes speech is usually underrepresented in, if not absent from speech corpora. Auxiliary input (learned accent embeddings, intermediate wav2vec2.0 representations) can address the decreased ASR recognition on this type of speech; the challenge remains that of achieving good accent conversion while preserving source speaker voice characteristics. The current approach builds on a pre-existing ACM by Jin et al. (2023) -- see above -- and aims to provide synthetic ASR training data using it. Phonetic knowledge is crucially injected into training to improve accent-specific pronunciation, and learnable accent representations are introduced to allow for variable accent strengths and adaptability to unseen accents.

The experimental setup involved evaluating two ASR models using Librispeech data. The first model (Base) utilized an efficient memory transformer followed by a recurrent neural transducer (RNNT), while the second model (HuBERT) had a similar structure with adjustments in channel configurations and dropout probabilities. The ASR models were tested on Librispeech data and accents from L2-Arctic corpus and Accented Vox Populi (AVP) dataset.

In experiments, the baseline ASR systems were trained without synthetic accented speech data, then evaluated. Three additional ASR models were trained with a combination of real and synthetic accented data, using a ratio of 80% real and 20% synthetic data. The ratio remained consistent across all accents. Finally, learned accent embeddings from L2-Arctic samples were visualized using t-SNE plots to assess their suitability for encoding accent information in an Accent Conversion Model (ACM).

RQ: Is it possible to improve ASR of accented speech with synthetic samples of a particular accent?

Results: The inclusion of one synthetic accent during ASR training had a positive effect on recognition results for that particular accent, which was a clear indicator that the ACM was able to synthesize a sufficient degree of accentedness. At the same time, HuBERT'd performance decreased with the use of synthetic data, likely due to the fact that it was not pre-trained on any and fine-tuning did not do enough. The Base model, which was trained from scratch, had a much grater benefit from the synthetic data. Notably, even when all seven accents were introduced in training, this did not improve performance on other unseen accents.

Overall, including one synthetic accent improved performance on that accent; and including several accents improved performance on those accents, but none of the conditions improved recognition on accents not seen in training. Additionally, pre-trained HuBERT did not benefit much from additional synthetic data fine-tuning, whereas a model trained from scratch saw much greater benefit from this approach.

Critical observations: Again, none of this replicable because the code is not available. It would have been also interesting to see a bit more ASR models be tested on this; this particular comparison does highlight the pre-trained/trained from scratch distinction in performance on this task, but there are other models that are seemingly good candidates and were not included.

Relevance: The authors show the potential for using synthetically accented data as a data augmentation approach to improve ASR performance on foreign-accented speech.

General insights

The synthesis of accented speech as a data augmentation method in ASR is promising for improving recognition performance on non-native speech. The three articles reviewed provide valuable insights into accent conversion methods and their implications for ASR systems. Zhao et al. (2018) shows the effectiveness of phonetic posteriograms in converting foreign-accented speech to sound more native-like and successfully decouples accent-related features from other speech characteristics. Jin et al. (2023) proposed a zero-shot multiple accent conversion approach, maintaining audio quality and speaker identity during conversion, albeit with limitations in accent classification for unfamiliar listeners. Klumpp et al. (2023) extended this work by integrating synthetic accented speech data into ASR training, showing improvements in recognition performance on the trained accents. However, the effectiveness varied depending on the model architecture, with pre-trained models benefiting less from synthetic data than models trained from scratch. Despite promising results, the lack of code availability and limited generalizability to unseen accents pose challenges for broader adoption. Overall, while accent conversion models offer a promising strategy for data augmentation in ASR, further research should focus on generalization and replicability for real-world applications.

References

Jin, M., Serai, P., Wu, J., Tjandra, A., Manohar, V., & He, Q. (2023, June). Voice-preserving zero-shot multiple accent conversion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

Klumpp, P., Chitkara, P., Sarı, L., Serai, P., Wu, J., Veliche, I. E., ... & He, Q. (2023). Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition. arXiv preprint arXiv:2303.00802.

Zhao, G., Sonsaat, S., Levis, J., Chukharev-Hudilainen, E., & Gutierrez-Osuna, R. (2018, April). Accent conversion using phonetic posteriorgrams. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5314-5318). IEEE.

Accent Modification

Introduction

Accents play a crucial role in shaping the unique characteristics of speech, reflecting an individual's linguistic background and cultural identity. However, the presence of foreign accents can sometimes pose challenges, particularly in the speaking test for language proficiency assessment.

Finkelstein, L., Zen, H., Casagrande, N., Chan, C., Jia, Y., Kenter, T., Petelin, A., Shen, J., Wan, V., Zhang, Y., Wu, Y., & Clark, R. (2022). Training Text-To-Speech Systems From Synthetic Data: A Practical Approach For Accent Transfer Tasks. Google LLC. Retrieved from https://arxiv.org/abs/2208.13183

Summary: This paper presents a practical approach for accent transfer tasks in text-to-speech (TTS) synthesis, where aspects of one speaker's speech are transferred to another speaker's speech. The authors address the challenge of creating high-quality transfer models that are also stable and suitable for user-facing applications. They propose a two-step training process involving a Tacotron-based accent transfer model and a robust CHiVE-BERT TTS system. The CHiVE-BERT system is trained on synthetic data generated by the Tacotron model, which results in high-quality audio with transferred accents while preserving speaker characteristics.

RQ: How can text-to-speech systems be trained to achieve accent transfer effectively and stably, without compromising the quality or usability of the synthesized speech?

Hypothesis: By training a robust TTS system on synthetic data generated by a less stable but high-quality accent transfer model, it is possible to achieve a balance between quality and stability in accent transfer tasks.

Conclusion: The study concludes that the proposed two-step training approach, using synthetic data generated by a Tacotron-based model to train a CHiVE-BERT system, yields reliable performance in terms of naturalness and accent transfer capability. The quality loss associated with the switch to synthetic data is within acceptable bounds, and the final system produces high-quality audio that maintains the original speakers' characteristics.

Critical observations: The authors note that the quality of the final system is affected by the intermediate Tacotron model, with some accents showing significant quality loss, particularly for female speakers in British English. Training on synthetic data can result in lower quality loss compared to using human recordings, possibly due to the reduced variance in synthetic data. The choice of vocoder, synthesizer, and the balance between synthetic and human recordings are critical in the training process, with the final system benefiting from a combination of both.

Relevance: The research on accent transfer in TTS systems aligns closely with my focus on accent modification for Turkish immigrants in Dutch oral exams. The methodologies explored for synthesizing and transferring accents can be adapted to develop tools that neutralize accents, enhancing exam fairness by ensuring evaluations are based on language skills rather than accent.

Li, W., Tang, B., Yin, X., Zhao, Y., Li, W., Wang, K., Huang, H., Wang, Y., & Ma, Z. (2020). Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech. arXiv preprint arXiv:2005.09271. Retrieved from https://arxiv.org/abs/2005.09271

Summary: This paper presents an end-to-end accent conversion framework aimed at transforming non-native accents into native accents while preserving the speaker's voice timbre. The proposed system introduces reference encoders to utilize multi-source information and optimizes the model architecture using GMM-based attention for improved synthesized performance. Experimental results show significant improvements in acoustic quality and native accent while retaining the non-native speaker's voice identity.

RQ: How can accent conversion be improved to better transform non-native accents into native accents in a way that maintains the original speaker's voice identity?

Hypothesis: Incorporating reference encoders and optimizing the model architecture with GMM-based attention will enhance the quality and naturalness of converted speech, leading to more effective accent conversion.

Conclusion: Incorporating reference encoders and optimizing the model architecture with GMM-based attention will enhance the quality and naturalness of converted speech, leading to more effective accent conversion.

Critical observations: The paper highlights the importance of prosodic and expressive information in accent conversion, which is effectively captured by the reference encoder. The GMM-based attention mechanism is found to be more stable and powerful for feature representation compared to traditional windowed attention.

Relevance: The research is relevant to accent modification efforts, particularly in language learning and pronunciation training contexts. The proposed accent conversion techniques could be applied to develop tools that help non-native speakers improve their pronunciation and reduce their accents, thereby enhancing communication and integration in societies where the target language is spoken natively.

Zang, X., Weng, F., & Zang, X. (2022). Foreign Accent Conversion using Concentrated Attention. In 2022 IEEE International Conference on Knowledge Graph (ICKG). Retrieved from https://ieeexplore.ieee.org/document/978-1-6654-5101-7

Summary: This paper introduces a novel method for foreign accent conversion (FAC) utilizing Phonetic Posteriorgrams (PPGs) and Log-scale Fundamental frequency (Log-FO) to address phonetic and prosody mismatches. The proposed approach employs concentrated attention to enhance the alignment of input sequences and mel-spectrograms, selecting the top k highest score values in the attention matrix row by row. The method is evaluated through objective metrics and demonstrates improved voice naturalness, speaker similarity, and accent similarity.

RQ: How can foreign accent conversion be improved to achieve better alignment and naturalness in synthesized speech while preserving the source speaker's identity?

Hypothesis: Implementing concentrated attention in the foreign accent conversion process will result in more accurate alignment of input sequences with mel-spectrograms, leading to improved accent conversion quality and naturalness in synthesized speech.

Conclusion: The proposed method using concentrated attention for foreign accent conversion delivers comparable or better results than previous methods in terms of voice naturalness and accent similarity. The concentrated attention mechanism effectively focuses on the most relevant frames for better alignment and synthesized speech quality.

Critical observations: The concentrated attention mechanism is found to be beneficial for achieving better alignment between input sequences and target sequences, resulting in improved speech synthesis.

Relevance: The research is relevant to the field of speech synthesis and voice conversion, particularly for applications that require the alteration of accents while maintaining the original speaker's voice characteristics. This work contributes to the development of systems that can aid in language learning, dubbing, and other scenarios where accent modification is beneficial, enhancing the quality and naturalness of synthesized speech.

Speech Separation

Zegers, J. (2019). CNN-LSTM models for multi-speaker source separation using Bayesian hyperparameter optimization. arXiv preprint arXiv:1912.09254.

Summary: This paper explores the use of Bayesian hyperparameter optimization for parallel CNN-LSTM models in the task of multi-speaker source separation (MSSS). Experiments were conducted with mixtures from the WSJ0 corpus and found that parallel CNN-LSTM models performed better than individual CNN or LSTM models.

Research Question (RQ): How does Bayesian hyperparameter optimization affect the performance of parallel CNN-LSTM models in multi-speaker source separation?

Hypothesis: The hypothesis was that the Bayesian optimization technique would find a better hyperparameter set that allows the parallel CNN-LSTM model to outperform individual CNNs or LSTMs in MSSS.

Conclusion: The study concluded that models with more trainable parameters in the LSTM portion performed better and that parallel CNN-LSTM models with Bayesian hyperparameter optimization outperformed the other models tested.

Critical Observations: The LSTM part of the model was crucial for performance, and bidirectional LSTMs performed better than unidirectional ones. Also, the study noted that more trainable parameters in the LSTM were generally preferred.

Relevance: This research is relevant for advancements in speech processing, specifically in improving source separation techniques which is a foundational task in many audio processing applications.

Isik, Y., Roux, J. L., Chen, Z., Watanabe, S., & Hershey, J. R. (2016). Single-channel multi-speaker separation using deep clustering. arXiv preprint arXiv:1607.02173

Summary: This study improved the baseline system for speaker-independent multi-speaker separation using deep clustering with an end-to-end signal approximation objective. By optimizing the model with enhancements like regularization, larger temporal context, and a deeper architecture, significant improvements in signal-to-distortion ratio and word error rate were achieved.

Research Question (RQ): Can the performance of speaker-independent multi-speaker separation be improved by using deep clustering with an end-to-end training approach?

Hypothesis: The authors hypothesized that incorporating an end-to-end signal approximation objective would lead to better performance in speech separation.

Conclusion: The paper concluded that the deep clustering approach with an end-to-end signal approximation objective greatly improved signal quality metrics and reduced speech recognition error rates, contributing to solving the cocktail party problem.

Critical Observations: The model performed well even with different numbers of speakers, and the addition of a signal approximation objective substantially reduced the word error rate when integrated with automatic speech recognition systems.

Relevance: This research contributes to solving complex audio environments' speech recognition challenges, aiding the development of better voice-activated systems that can function effectively in real-world conditions.

Maiti, S., Ueda, Y., Watanabe, S., Zhang, C., Yu, M., Zhang, S., & Xu, Y. (2023). EEND-SS: Joint end-to-end neural speaker diarization and speech separation for flexible number of speakers. In 2022 IEEE Spoken Language Technology Workshop (SLT) (pp. 480-487). IEEE.

Summary: The paper presents EEND-SS, a framework that integrates speaker diarization, speech separation, and speaker counting into a single end-to-end trainable model. It demonstrated improved performance over single-task models and enhanced speaker counting for a flexible number of speakers.

Research Question (RQ): Can an integrated framework that combines speaker diarization and speech separation improve performance over models that address these tasks separately?

Hypothesis: The authors posited that a joint model incorporating speaker diarization, speech separation, and speaker counting would perform better than individual models tackling each task separately.

Conclusion: The study concluded that the EEND-SS framework could outperform single-task baselines in both diarization and separation metrics and improved speaker counting performance.

Critical Observations: A key observation was that jointly learning to separate and diarize helped the model perform better in diarization, particularly in less overlapped conditions, suggesting better generalization.

Relevance: The results of this study are highly relevant for multi-speaker environments, improving the performance and applicability of voice recognition systems in scenarios with a variable number of speakers. Each of these studies contributes to the field of speech processing, advancing our understanding and capability in separating and recognizing speech in challenging audio scenarios.

Speech Synthesis Evaluation

Le Maguer, S., King, S., & Harte, N. (2024). The limits of the Mean Opinion Score for speech synthesis evaluation. Computer Speech & Language, 84, 101577. https://doi.org/10.1016/j.csl.2023.101577

Summary: The paper critically evaluates the Mean Opinion Score (MOS) as an evaluation metric of synthetic speech. The authors conduct 4 experiments related to the Blizzard Challenge to assess the stability and reliability of MOS, the influence of varying quality systems on MOS, and how the introduction of modern technologies affects the scoring of historical systems.

Research Question (RQ): How reliable and stable is the Mean Opinion Score (MOS) when used for speech synthesis evaluation, especially with modern speech synthesis technologies that closely approximate human speech?

Hypothesis: MOS, despite being a standard evaluation metric, is a relative score influenced by the presence of both lower and higher quality systems in the evaluation set and may not adequately reflect the advancements in modern speech synthesis technologies.

Conclusion: The study concludes that MOS is influenced by the relative quality of the systems being evaluated and suggests that MOS has reached its limits in terms of effectiveness for evaluating modern speech synthesis technologies. New evaluation protocols that better capture the nuances of current systems are needed.

Critical Observations: The authors observe that MOS tends to be relative rather than absolute, its scores can vary over time, and it is sensitive to the presence of anchors. The presence of high-quality modern systems can influence the MOS of historical systems, often leading to a compression of scores.

Relevance: This research is relevant for the field of speech synthesis evaluation, particularly as the technology has reached a quality close to human speech. It challenges the current predominant reliance on MOS and argues for the development of more sophisticated evaluation protocols that can better analyze modern synthesis technologies.

O’Mahony, J., Oplustil-Gallegos, P., Lai, C., & King, S. (2021). Factors Affecting the Evaluation of Synthetic Speech in Context. 11th ISCA Speech Synthesis Workshop (SSW 11), 148–153. https://doi.org/10.21437/SSW.2021-26

Summary: The paper examines factors that influence the evaluation of synthetic speech in context, particularly as Text-to-Speech (TTS) synthesis approaches naturalness limits for isolated sentences. It explores the effect of instructions given to participants, the impact of between-sentence textual context dependency, and the sensitivity of Mean Opinion Score (MOS) to prosodic differences in synthetic speech.

Research Question (RQ): How do various factors such as listener instructions, between-sentence textual context dependency, and prosodic realizations of synthetic speech affect the evaluation of synthetic speech in context?

Hypothesis: The authors hypothesize that the wording of instructions given to listeners, the textual context of sentences, and the prosody of synthetic speech can significantly affect the MOS ratings, potentially causing variations in the assessment of speech synthesis quality.

Conclusion: The study finds that listener instructions significantly impact MOS ratings, with 'appropriateness' and 'naturalness' being interpreted differently. Textual context dependency does not significantly affect ratings, and listeners are sensitive to prosodic differences. The MOS is an appropriate paradigm for evaluating prosodic differences in synthetic speech.

Critical Observations: The authors observe that despite non-context-aware synthesis, utterances presented in context receive higher MOS ratings than those in isolation. Furthermore, participants' interpretation of 'appropriateness' contributes to higher ratings in context, and MOS ratings are sensitive to substantial prosodic differences.

Relevance: This research is relevant for advancing TTS evaluation methods. It suggests that the MOS rating system needs to consider the influence of contextual factors and prosody for long-form speech synthesis evaluation, indicating a shift from traditional sentence-level assessment paradigms.

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Finkelstein et al.(2022): Chenyu Li
• Article Li et al.(2020): Chenyu Li
• Article Zang et al(2022): Chenyu Li
• Article Grimm et al.(2007): Yining Lei
• Article Z.Huang et al.(2014):Siqi Zheng
• Introduction: Chenyu Li
• Synthesis: All

Subsections:

The section Synthetically improving foreign-accented speech recognition was written by Maria Tepei.

The section Accent Modification was written by Chenyu Li

The section Speech Separation was written by Sherry Yu-Ting Yeh

The section Speech Synthesis Evaluation was written by Brandi Hongell

ASR III

Introduction

Briefly introduce your thematic focus and its significance in the field of speech technology.

Article summaries

Kartik, A., Andrew, R., Abhinav, S., Bhu-vana, R., & Brian, K. (2017, March). End-to-end ASR-free keyword search from speech. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing.

• Summary: The paper introduces an end-to-end ASR-free system for keyword search (KWS) from speech, which leverages minimal supervision. The system comprises three sub-systems: an RNN-based acoustic auto-encoder, a CNN-RNN character language model, and a feed-forward neural network for KWS. This architecture eliminates the need for conventional ASR systems and transcription of audio data, enabling faster training and performance that rivals traditional methods.
• RQ: The main research question explored is whether an end-to-end ASR-free system can effectively perform text query-based keyword search from speech with minimal supervision, and how its performance compares to traditional ASR-based systems.
• Hypothesis: The hypothesis posited is that an end-to-end ASR-free keyword search system, despite not utilizing a conventional ASR system or fully transcribed training audio, can still achieve respectable performance in identifying keywords within speech utterances.
• Conclusion: The ASR-free E2E KWS system demonstrated the ability to perform keyword search tasks with minimal supervision, achieving respectable results compared to a conventional hybrid HMM-DNN ASR system but with significantly reduced training time. This system represents a promising direction for efficient and scalable KWS from speech without relying on comprehensive transcription data or traditional ASR systems.
• Critical observations:
  • The E2E system's performance on in-vocabulary (IV) and out-of-vocabulary (OOV) queries is noteworthy, especially for OOV queries where it slightly outperforms the hybrid ASR system.
  • The system's performance is limited for shorter queries, indicating challenges in capturing reliable representations for queries lacking context.
  • The efficiency in training time (36 times faster than traditional methods) without substantial loss in accuracy points to the potential for scalability and application in low-resource settings.
• Relevance: This work has significant implications for the field of speech recognition and information retrieval, especially in environments where rapid deployment and adaptation are critical. By demonstrating that an ASR-free approach can yield comparable performance to more traditional, labor-intensive systems, this research opens up new possibilities for keyword search applications in multilingual and resource-constrained scenarios.

Zarazaga, P. P., Henter, G. E., & Malisz, Z. (2023, June). A processing framework to access large quantities of whispered speech found in ASMR. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

• Summary: This paper introduces a novel processing framework to harness large volumes of high-quality whispered speech from ASMR content. By employing an advanced whispered activity detection (WAD) system and integrating human-in-the-loop through Edyson, a bulk audio-annotation tool, the framework efficiently labels and extracts clean whispered speech segments. The approach not only aids in the development of whisper-capable speech technology but also contributes valuable linguistic data for research.
• RQ: The research question addressed by the paper is how to effectively process and extract large amounts of clean whispered speech from ASMR recordings, which include a variety of background noises and non-whispered acoustic triggers.
• Hypothesis: The hypothesis posited in the paper is that by leveraging sophisticated WAD techniques, coupled with human-in-the-loop annotation and data augmentation, it is possible to efficiently identify and isolate high-quality whispered speech segments from the complex acoustic landscape of ASMR content.
• Conclusion: The framework presented successfully processes ASMR recordings to access and extract significant amounts of clean whispered speech, outperforming traditional methods. This success opens up new avenues for speech technology development and linguistic research, particularly in fields requiring large datasets of natural whispered speech.
• Critical observations:
  • The paper highlights the scarcity of whispered speech datasets and the challenges in processing ASMR content due to its diverse acoustic triggers.
  • The use of deep learning for whispered activity detection significantly improves the accuracy of identifying whispered segments within noisy environments.
  • Incorporating human judgment through Edyson for audio labeling enhances the precision of the extracted data, making the process more efficient and scalable.
• Relevance: The research is highly relevant to advancing speech recognition technologies, especially for applications requiring whispered input. It also provides a substantial resource for studying the linguistic and acoustic properties of whispered speech, potentially impacting areas like human-computer interaction, where natural and nuanced speech inputs are increasingly important.

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

• End-to-end ASR-free keyword search from speech: Patrick OUYANG
• A processing framework to access large quantities of whispered speech found in ASMR: River LIN