Editing State-of-the-art (section)

=== Article summaries ===

==== Wang, S., Rohdin, J., Plchot, O., Burget, L., Yu, K., & Cernocky, J. (2020). Investigation of Specaugment for Deep Speaker Embedding Learning. ''ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'', 7139–7143. <nowiki>https://doi.org/10.1109/ICASSP40776.2020.905348</nowiki> ====
* Summary: The article investigates the effectiveness of SpecAugment, a data augmentation method, for speaker verification tasks using TDNN and ResNet34 models with Softmax and AAMSoftmax loss functions. Experiments on NIST SRE 2016 Cantonese and Tagalog subsets and Voxceleb1 dataset show improved performance with SpecAugment, achieving 3.72% and 11.49% EER for NIST SRE 2016 Cantonese and Tagalog, respectively, and 1.47% EER for Voxceleb1. SpecAugment demonstrates promising results for speaker verification across different languages, enhancing system robustness without complex offline augmentation.
* RQ: How effective is SpecAugment, a data augmentation method originally proposed for speech recognition, when applied to speaker verification tasks across different languages, specifically Cantonese and Tagalog?
* Hypothesis: Applying SpecAugment, a data augmentation technique initially developed for speech recognition, to speaker verification tasks will lead to performance improvements across different languages, including Cantonese and Tagalog.
* Conclusion: Implementing SpecAugment for speaker verification tasks yields significant performance improvements across different languages. Specifically, the study demonstrates that SpecAugment, applied on-the-fly without complex offline augmentation methods, achieves state-of-the-art results in speaker verification tasks for Cantonese and Tagalog, as well as for the Voxceleb1 dataset.
* Critical observations: The critical observation of the article focuses on the implementation of SpecAugment for speaker verification tasks across various languages, particularly Cantonese and Tagalog, which are considered low-resource languages. The study demonstrates that SpecAugment, applied on-the-fly, effectively improves performance in speaker verification tasks for these languages, achieving significant reductions in Equal Error Rate (EER) compared to traditional methods. This highlights the potential of SpecAugment as a simple yet powerful augmentation technique, particularly beneficial for low-resource language processing tasks.
* Relevance: The relevance of the article to the topic of low-resource language Automatic Speech Recognition (ASR) lies in its exploration of SpecAugment as a data augmentation technique for speaker verification tasks in languages like Cantonese and Tagalog, which are considered low-resource. By demonstrating the effectiveness of SpecAugment in improving performance in speaker verification tasks for these languages, the study showcases a potential solution to the challenges posed by limited data availability in low-resource language ASR. This highlights SpecAugment as a valuable tool for enhancing ASR systems' robustness and accuracy in under-resourced linguistic contexts.

==== Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., ... & Wu, Y. (2023). Google USM: Scaling automatic speech recognition beyond 100 languages. ''arXiv preprint arXiv:2303.01037''. ====

* Summary: Google's Universal Speech Model aims to develop an ASR model that will be able to perform speech recognition on all languages of the world. This paper leverages large amounts of unlabelled speech and text data from YouTube to train a multilingual-encoder that can then be used in fine-tuning on very small amounts of labelled data. This allows them to outperform Whisper<ref>Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023, July). Robust speech recognition via large-scale weak supervision. In ''International Conference on Machine Learning'' (pp. 28492-28518). PMLR.</ref> with significantly less labelled data, while also showing that this approach works positively for lower-resource languages.
* RQ: Can we leverage the large amounts of unlabelled speech data to perform massively multilingual ASR and speech translation?
* Hypothesis: By using a vast amount of unlabelled data, the encoder will learn speech representations that can be leveraged in fine-tuning and downstream tasks.
* Conclusion: Pre-training on unlabelled data is an effective way to improve multilingual performance while requiring much less labelled data.
* Critical observations: Although they keep mentioning that their performance is stellar on low-resource languages, no results were presented for these languages specifically. Most results are from multilingual datasets that might be imbalanced as well. Furthermore, the models and training data are not publicly available, making the research less approachable for improvements.
* Relevance: This paper is highly relevant for our theme as it aims to improve low-resource ASR through unlabelled data, which is an effective solution to the data scarcity problem.

==== Zhang, Y., Herygers, A., Patel, T., Yue, Z., & Scharenborg, O. (2023). ''Exploring data augmentation in bias mitigation against non-native-accented speech'' (arXiv:2312.15499). arXiv. <nowiki>http://arxiv.org/abs/2312.15499</nowiki> ====

* Summary: The study aimed to investigate the impact of data augmentation techniques on the performance of Flemish Automatic Speech Recognition (ASR) systems for both native Flemish speakers and those with non-native accents. Specifically, the research focused on addressing biases against non-native-accented Flemish speech. Various data augmentation methods were applied to augment the training data, and the performance of the ASR system was evaluated using both native and non-native speakers' speech samples. The results suggested that tailored data augmentation techniques can lead to improved ASR system performance for both native and non-native-accented Flemish speech. This finding highlights the potential of data augmentation in mitigating bias and enhancing the accuracy of ASR systems across diverse speaker demographics.
* RQ: What is the optimal type of data augmentation, in terms of reducing bias against non-native-accented Flemish in a Flemish ASR system, when applied to both native Flemish and non-native-accented Flemish?
* Hypothesis: Applying specific types of data augmentation techniques, tailored to address bias against non-native-accented Flemish speech, will lead to improved performance in a Flemish Automatic Speech Recognition (ASR) system for both native Flemish and non-native-accented Flemish speakers.
* Conclusion: The study concluded that employing tailored data augmentation techniques can significantly improve the performance of Flemish Automatic Speech Recognition (ASR) systems, particularly in mitigating biases against non-native-accented speech. By augmenting the training data with techniques specifically designed to address the characteristics of non-native accents, the ASR system demonstrated notable enhancements in accuracy for both native and non-native speakers. These findings underscore the importance of considering diversity in training data and utilizing appropriate augmentation strategies to enhance the robustness and inclusivity of ASR systems.
* Critical observations: The performance of Flemish Automatic Speech Recognition (ASR) systems can be significantly improved through the use of tailored data augmentation techniques. Specifically, augmenting the training data with methods designed to address the characteristics of non-native accents resulted in notable enhancements in accuracy for both native and non-native speakers. This observation highlights the importance of considering diversity in training data and employing appropriate augmentation strategies to enhance the inclusivity and robustness of ASR systems.
* Relevance:  Low-resource languages often suffer from limited available data for training ASR systems, which can lead to poor performance, especially for speakers with non-native accents. This study demonstrates that tailored data augmentation techniques can substantially improve the accuracy of ASR systems, even in scenarios with limited training data.By addressing the challenges faced by speakers with non-native accents, the paper contributes valuable insights into how ASR technology can be adapted and optimized for low-resource languages. It underscores the importance of developing strategies that account for linguistic diversity and accent variations, ultimately making ASR systems more inclusive and effective in diverse linguistic contexts. Therefore, the findings of this study are highly relevant for researchers and practitioners working on ASR for low-resource languages, offering practical approaches to enhance system performance and usability in such settings.

==== Wang, H., Wang, S., Zhang, W. Q., & Bai, J. (2023). Distilxlsr: A light weight cross-lingual speech representation model. ''arXiv preprint arXiv:2306.01303''. ====

*Summary: The authors introduce a compression scheme for multilingual self-supervised speech representation models aimed at enhancing speech recognition performance for low-resource languages while reducing model size for industrial applications. Experiments across two types of teacher models and 15 low-resource languages demonstrate that this method can reduce parameters by 50% while maintaining cross-lingual representation capabilities. The approach is shown to be generalizable across various languages and teacher models, with potential to improve the cross-lingual performance of English pretrained models. Key observations include the effectiveness of data splicing, the importance of layer-jumping initialization, the balance between model compression and performance, and underfitting challenges in low-resource scenarios.
* RQ: The paper investigates how to compress multilingual self-supervised speech representation models, specifically aiming to enhance speech recognition performance for low-resource languages while reducing the model size for easier industrial application.
* Hypothesis: It's possible to significantly reduce the size of multilingual speech representation models without substantially sacrificing performance across various languages by distilling cross-lingual models using only English data and applying techniques such as random phoneme shuffling, layer-jumping initialization, and data splicing.
* Conclusion: The proposed DistilXLSR model successfully reduces parameter size by 50% while maintaining cross-lingual representation capabilities across 15 low-resource languages. This model demonstrates its effectiveness through experimental results, showing comparable performance to larger teacher models and the potential for generalizability and improvement in cross-lingual performance of English pre-trained models.
* Critical Observations:
*# Randomly shuffling syllables within utterances to reduce linguistic information proved effective for distilling models with cross-lingual capabilities using only English data.
*# This novel method of initializing student models by leveraging teacher models' pre-trained weights across layers enhances the learning and representation ability of the distilled model.
*# The study highlights a trade-off between model size and performance, where the distilled models show only slight degradation in performance despite a significant reduction in size.
*# The paper acknowledges challenges like underfitting, especially evident in datasets with lower quality audio, suggesting that further research could explore structured pruning or other methods to mitigate this.
* Relevance: By employing a novel distillation approach that leverages English data, this model addresses the challenge of accessing and formatting training data across multiple languages, which is particularly difficult for low-resource languages. The effectiveness of DistilXLSR in maintaining performance across 15 low-resource languages, despite a substantial reduction in model size, showcases its potential in breaking down language barriers and enabling more equitable access to speech technology worldwide.

==== Gandhi, S., von Platen, P., & Rush, A. M. (2023). Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling. ''arXiv preprint arXiv:2311.00430''. ====

* Summary: The study introduces a novel approach to compressing pre-trained large speech recognition models for efficient deployment in low-resource environments. By leveraging large-scale pseudo-labeling, the research achieves a smaller variant, Distil-Whisper, which significantly reduces the model size and inference time without considerably sacrificing performance. This method particularly benefits low-resource languages by maintaining robustness across various acoustic scenarios and demonstrating potential in extending sophisticated ASR capabilities to languages with limited training data.
* RQ: How can the size of pre-trained speech recognition models, specifically the Whisper model, be reduced for efficient deployment in low-latency or resource-constrained environments while maintaining model robustness and performance?
* Hypothesis: By using pseudo-labelling to create a large-scale open-source dataset and applying a simple word error rate (WER) heuristic to select only the highest quality pseudo-labels for training, it is possible to distill the Whisper model into a smaller variant (Distil-Whisper) that is significantly faster and more parameter-efficient without substantially sacrificing performance.
* Conclusion: Distil-Whisper successfully demonstrates the feasibility of distilling a large-scale speech recognition model into a significantly smaller and faster version without substantial loss in performance. The distilled model achieves a WER performance within 1% of the original Whisper model on out-of-distribution test data, maintains robustness against difficult acoustic conditions, and reduces the propensity for hallucination errors in long-form audio. Furthermore, Distil-Whisper, when paired with Whisper for speculative decoding, offers a significant speed-up in inference times while ensuring identical outputs to the original model.
* Critical Observations: The approach underscores the effectiveness of large-scale pseudo-labelling and a straightforward WER-based heuristic in filtering training data for distillation purposes. The research highlights a crucial balance between model size, speed, and performance robustness, contributing to practical speech recognition applications, especially in constrained environments.
* Relevance: The methodology demonstrates potential for extending sophisticated ASR capabilities to languages with fewer resources by leveraging transfer learning and pseudo-labeling techniques.

==== Yang, M., Tjandra, A., Liu, C., Zhang, D., Le, D., & Kalinli, O. (2023, June). Learning asr pathways: A sparse multilingual asr model. In ''ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'' (pp. 1-5). IEEE. ====

* Summary: This research proposes a sparse multilingual ASR model, ASR pathways, which employs language-specific sub-networks to effectively manage multilingual speech recognition without significant performance drops in low-resource languages. The model utilizes iterative magnitude pruning (IMP) and the Lottery Ticket Hypothesis (LTH) to learn language-specific masks, facilitating knowledge transfer and improved performance in languages with scant data. This method enhances the accessibility of advanced ASR technologies in multilingual contexts, showing promise in scaling speech recognition capabilities across diverse language landscapes, including those with fewer resources.
* RQ: How can neural network pruning be optimized for multilingual Automatic Speech Recognition (ASR) without significantly degrading recognition performance on certain languages, given that language-agnostic pruning may discard important language-specific parameters?
* Hypothesis: It's possible to construct a sparse multilingual ASR model, referred to as ASR pathways, which activates language-specific sub-networks or "pathways" for different languages. This approach enables both language-specific optimization and the shared learning of parameters across languages, particularly benefiting lower-resource languages through joint multilingual training.
* Conclusion: The ASR pathways model, which utilizes sparse sub-networks tailored for specific languages within a unified parameter set, outperforms both dense models and language-agnostically pruned models. It demonstrates improved performance on low-resource languages compared to monolingual sparse models, showcasing the effectiveness of this sparse multilingual ASR framework in achieving efficient and robust speech recognition across multiple languages.
* Critical observations: The study found that language-specific pruning masks, developed through Iterative Magnitude Pruning (IMP) or Lottery Ticket Hypothesis (LTH), are crucial for the model's success. These masks enable the model to maintain or even improve performance across languages by preserving essential language-specific parameters while also benefiting from shared knowledge. The LTH approach, in particular, showed superior performance, even with fewer total effective parameters, highlighting the importance of the initial parameter selection in the pruning process.
* Relevance: The shared parameters between these language-specific pathways facilitate knowledge transfer during joint multilingual training, which is especially beneficial for languages with limited training data. The empirical results showing improved performance on low-resource languages compared to monolingual sparse models underline the potential of this method to bring high-quality ASR technologies to low-resource settings.

==== N, K. D., Wang, P., & Bozza, B. (2021). Using Large Self-Supervised Models for Low-Resource Speech Recognition. ''Interspeech 2021'', 2436–2440. <nowiki>https://doi.org/10.21437/Interspeech.2021-631</nowiki> ====
* Summary: This paper investigates the effectiveness of using large self-supervised pre-trained models (such as wav2vec 2.0) for low-resource speech recognition tasks. The authors conducted experiments on three Indian languages (Telugu, Tamil, and Gujarati), using different pre-trained models (monolingual English, multilingual) and compared different fine-tuning strategies (CTC, seq2seq, etc.).
* RQ: For low-resource speech recognition tasks, how effective are large self-supervised pre-trained models (such as wav2vec 2.0) compared to traditional supervised learning methods? For Indian languages, are cross-lingual multilingual pre-trained models or monolingual English pre-trained models more suitable? How do different fine-tuning strategies (CTC vs seq2seq) affect model performance? Additionally, how well do these pre-trained models generalize to seen and unseen languages?
* Hypothesis:
*# Large self-supervised pre-trained models will outperform supervised learning models under low-resource conditions.
*# Cross-lingual multilingual pre-trained models will perform better than monolingual English models on these Indian languages.
*# Adopting the CTC fine-tuning strategy will achieve better performance than the seq2seq strategy.
* Conclusion:The multilingual pre-trained model XLSR outperformed the monolingual models on all three languages; for seen languages (like Tamil), the pre-trained model can approach the best performance with only 50% of the training data; the CTC fine-tuning framework performed better than the seq2seq framework, possibly due to the small amount of data; even smaller English pre-trained models showed decent transfer performance on Indian languages.
* Critical observations:The authors did not explain why the larger English pre-trained model underperformed compared to the smaller one, and analysis of the multilingual fine-tuning strategy was limited, only compared to the monolingual strategy. In addition, the impact of different pre-training corpora on model performance was not explored.
* Relevance:This work is important for low-resource speech recognition domains in developing countries. Leveraging large self-supervised pre-trained models can make full use of unlabeled data, alleviating the bottleneck of limited labeled data. This study provides an effective solution for low-resource speech recognition tasks.

==== Yi, C., Wang, J., Cheng, N., Zhou, S., & Xu, B. (2021). ''Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages'' (arXiv:2012.12121). arXiv. <nowiki>http://arxiv.org/abs/2012.12121</nowiki> ====
*Summary:  The authors applied the pre-trained wav2vec2.0 model to low-resource speech recognition across six languages. Despite being pre-trained on a different domain, wav2vec2.0 could effectively adapt when fine-tuned on limited transcribed speech, even outperforming supervised pre-training approaches. Using coarser modeling units like subwords/characters worked better than finer units like phonemes/letters. Critically, self-supervised pre-training on large unlabeled data enabled wav2vec2.0 to learn robust speech representations that transferred well across languages and domains, showcasing its impressive potential for tackling low-resource speech tasks.
*RQ: Can the pre-trained wav2vec2.0 model, which was trained on English audiobook data, be effectively applied to low-resource speech recognition tasks in various languages and real-world spoken scenarios?
* Hypothesis: The self-supervised pre-training of wav2vec2.0 allows it to learn general acoustic representations that can be adapted to different languages and domains, even with limited transcribed data.
* Conclusion:  The experiments demonstrate that wav2vec2.0 can achieve significant performance improvements on low-resource speech recognition tasks across six languages (Arabic, English, Mandarin, Japanese, German, and Spanish) compared to previous methods. The largest gain of 52.4% was observed for English, likely due to the pre-training data being in English. Using coarser-grained modeling units like subwords or characters generally performed better than finer-grained units like phones or letters.
* Critical observations:
*# Self-supervised pre-training on a large amount of unlabeled data from other languages can be more effective than supervised pre-training on limited target language data.
*# The encoder-decoder structure did not perform well in low-resource scenarios, possibly due to the decoder's inability to generalize from sparse transcriptions.
*# External language models provided significant performance gains across all languages, model sizes, and modeling units.

*Relevance:  This research highlights the potential of self-supervised pre-trained models like wav2vec2.0 to alleviate the data scarcity problem in low-resource speech recognition tasks. It demonstrates the model's ability to adapt to various languages and spoken domains, even when pre-trained on data from a different domain (audiobooks). The findings suggest that large-scale self-supervised pre-training can learn robust acoustic representations that can be effectively transferred to downstream tasks with limited data.

==== Thomas, B., Kessler, S., & Karout, S. (2022). ''Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition'' (arXiv:2202.03218). arXiv. <nowiki>http://arxiv.org/abs/2202.03218</nowiki> ====
*Summary:  In this paper the authors applied adapter modules to a pre-trained wav2vec 2.0 model in order to perform downstream ASR tasks such as multilingual speech recognition. Compared with full fine-tuning, inserting adapters shows benefits of reducing the number of parameters and increasing the scalability of the model.
*RQ: The authors asked if applying adapters on self-supervised ASR models would show the same benefits as in an NLP model.
* Hypothesis: The authors hypothesized that the wav2vec 2.0 model tuned with adapter modules would be able to perform downstream tasks with little performance degradation.
* Conclusion: Self-supervised speech models can be utilized in a more parameter-efficient manner without sacrificing performance. The monolingual model such as wav2vec 2.0  can be successfully adapted to a multilingual ASR model. The multilingual model that the authors trained themselves also demonstrated capabilities to recognize English or French.
* Critical observations: 
** Adapters perform slightly worse than fine-tuning on English ASR.
** French ASR saw a slight performance increase using adapters.
** Multilingual pre-trained models using adapters also get close performance as in fine-tuning.
** Adapters add only a small number of additional parameters per task.

*Relevance: This paper is the first paper that applies adapters on self-supervised ASR models. It provides insight on how adapters can be used as a quicker and computationally inexpensive method to tune the model for downstream tasks and multi-tasks. It is highly relevant to low-resource ASR because low-resource languages usually have less training data and are easy to overfit with a full fine-tuning approach. Adapter approach can prevent tuning model from overfitting.

==== Schultz, B.G., Tarigoppula, V.S.A., Noffs, G. ''et al.'' Automatic speech recognition in neurodegenerative disease. ''Int J Speech Technol'' 24, 771–779 (2021). <nowiki>https://doi-org.proxy-ub.rug.nl/10.1007/s10772-021-09836-w</nowiki> ====
*Summary: The paper evaluates the performance of three state-of-the-art automatic speech recognition (ASR) platforms (Amazon Web Services, Google Cloud, and IBM Watson) on speech from individuals with neurodegenerative diseases (multiple sclerosis and Friedreich's ataxia) and healthy controls.
* RQ: How well do commercial ASR systems perform on dysarthric speech from individuals with neurodegenerative diseases compared to healthy speech?
* Hypothesis: ASR accuracy will be lower for dysarthric speech from neurodegenerative disease groups compared to healthy controls, and accuracy will decline with increased disease severity and duration.
* Conclusion: ASR accuracy was significantly higher for healthy controls than clinical groups, and higher for multiple sclerosis compared to Friedreich's ataxia. Amazon Web Services and Google Cloud outperformed IBM Watson. Accuracy decreased with increased disease duration for Friedreich's ataxia but not multiple sclerosis. Age and sex did not significantly affect ASR accuracy.
* Critical observations: 
** ASR faces challenges in recognizing dysarthric speech from neurodegenerative diseases.
** Accuracy declines as consecutive words increase, irrespective of speech impairment.
** Severity of speech impairment, as indicated by disease type and duration, negatively impacts ASR accuracy.
* Relevance: The theme focuses on low-resource ASR for underrepresented languages. While this study does not directly address low-resource languages, it highlights the challenges ASR systems face in recognizing atypical speech patterns, which is relevant for low-resource languages with diverse speaker populations and dialects. Improving ASR performance on dysarthric speech could inform techniques for handling speech variability in low-resource settings.

==== Vásquez-Correa, J. C., Rios-Urrego, C. D., Arias-Vergara, T., Schuster, M., Rusz, J., Nöth, E., & Orozco-Arroyave, J. R. (2021). Transfer learning helps to improve the accuracy to classify patients with different speech disorders in different languages. ''Pattern Recognition Letters'', ''150'', 272–279. <nowiki>https://doi.org/10.1016/j.patrec.2021.04.011</nowiki> ====
*Summary: The paper proposes using transfer learning with convolutional neural networks (CNNs) to classify pathological speech from patients with neurodegenerative disorders like Parkinson's disease (PD) and Huntington's disease (HD). Time-frequency representations of voice onset/offset segments are used as input to the CNNs. Two transfer learning scenarios are explored: 1) transferring a model trained on one language to classify patients speaking a different language, and 2) transferring a model trained on one disorder (e.g. PD) to classify patients with a different disorder (e.g. HD).
* RQ: Can transfer learning improve the accuracy of CNN models for classifying pathological speech across different languages and disorders?
* Hypothesis: Transferring knowledge from a base model trained on one language/disorder to a target model for a different language/disorder can improve classification accuracy when there is limited data for the target task.
* Conclusion: The results suggest transfer learning can improve target model accuracy, but only when the base model is sufficiently accurate. Transferring between similar tasks (e.g. different languages) works better than transferring between very different tasks (e.g. different disorders).
* Critical observations:
** Accuracies ranged from 70-89% across languages without transfer learning
** Transferring between languages improved accuracy in some cases (e.g. Spanish -> German improved over training on German alone)
** Transferring between very different disorders like PD and HD did not improve over training directly on the target disorder
* Relevance: The paper does not directly address low-resource ASR, but instead focuses on pathological speech classification. However, some insights around transfer learning across languages could potentially be adapted to low-resource ASR scenarios.