State-of-the-art

Theme: Template copy/paste but do not delete

Introduction

Briefly introduce your thematic focus and its significance in the field of speech technology.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Jones et al. 2023: YOUR NAME
• Article XXX: YOUR NAME
• Introduction: All
• Synthesis: All

Low-resource ASR

Introduction

Our theme focuses on automatic speech recognition (ASR) of low-resource languages. Low-resource languages are often underrepresented in ASR due to the limited amount of data, limited amount of speakers, and low commercial impact. However, it is important for both preserving and encouraging the use of low-resource languages to allow for users to utilize ASR for their own language. Therefore, our theme is significant in the field of speech technology.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., ... & Wu, Y. (2023). Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037.

• Summary: Google's Universal Speech Model aims to develop an ASR model that will be able to perform speech recognition on all languages of the world. This paper leverages large amounts of unlabelled speech and text data from YouTube to train a multilingual-encoder that can then be used in fine-tuning on very small amounts of labelled data. This allows them to outperform Whisper with significantly less labelled data, while also showing that this approach works positively for lower-resource languages.
• RQ: Can we leverage the large amounts of unlabelled speech data to perform massively multilingual ASR and speech translation?
• Hypothesis: By using a vast amount of unlabelled data, the encoder will learn speech representations that can be leveraged in fine-tuning and downstream tasks.
• Conclusion: Pre-training on unlabelled data is an effective way to improve multilingual performance while requiring much less labelled data.
• Critical observations: Although they keep mentioning that their performance is stellar on low-resource languages, no results were presented for these languages specifically. Most results are from multilingual datasets that might be imbalanced as well. Furthermore, the models and training data are not publicly available, making the research less approachable for improvements.
• Relevance: This paper is highly relevant for our theme as it aims to improve low-resource ASR through unlabelled data, which is an effective solution to the data scarcity problem.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Jones et al. 2023: YOUR NAME
• Article Google USM: Scaling automatic speech recognition beyond 100 languages: Ömer Tarik
• Introduction: Ömer Tarik
• Synthesis: All

Language-specific Text-To-Speech

Introduction

Briefly introduce your thematic focus and its significance in the field of speech technology.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Jones et al. 2023: YOUR NAME
• Article XXX: YOUR NAME
• Introduction: All
• Synthesis: All

Theme: Non-Language-specific Text-To-Speech

Introduction

TTS systems have significantly advanced over time, achieving remarkable intelligibility and near-human naturalness in synthetic voices through deep learning advancements. However, the naturalness of synthetic voice remains limited to sentences, and lacks the expressivity found in human conversation such as appropriate emotion, prosody and style. Despite these limitations, natural TTS, particularly expressive speech synthesis, plays a crucial role in achieving human-like speech and enhancing the engagement of synthesized speech. Moreover, it facilitates the broader adoption of TTS technology across various domains within the field of speech technology. In this context, our group focuses on the theme of TTS naturalness with two interconnected subthemes: exploring advanced models and theoretical frameworks. By addressing these subthemes, we aim to provide a comprehensive overview of the current state-of-the-art in TTS naturalness.

Article summaries

Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., ... & Zhao, Z. (2023, July). Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning (pp. 13916-13932). PMLR.

• Summary: The article introduces "Make-An-Audio," a system utilizing a prompt-enhanced diffusion model for TTS generation, aiming to improve the naturalness and expressiveness of synthesized audio.
• RQ: How does the model improve the naturalness of TTS?
• Hypothesis: By introducing pseudo prompt enhancement and spectrogram autoencoders, the model can effectively utilize unsupervised language-free data and higher-level semantic understanding to enhance the naturalness and expressiveness of speech synthesis.
• Conclusion: "Make-An-Audio" successfully enhances the naturalness and expressiveness of speech synthesis, achieving state-of-the-art performance in evaluations.
• Critical observations: The performance of "Make-An-Audio" is still partly dependent on extensive data and complex model training. In addition, there is still space for improvement in expressing the emotions and rhythms of human conversations.
• Relevance: The "Make-An-Audio" system presented in the paper offers an effective solution to the limitations in naturalness and expressiveness currently faced by TTS.

APA Citation of NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality

• Summary: NaturalSpeech proposes a system for converting text to speech (TTS) that achieves human-level quality. It leverages a variational autoencoder (VAE) to bridge the gap between text and speech waveforms.
• RQ (Research Question): Can a TTS system achieve speech quality indistinguishable from humans?
• Hypothesis: By incorporating a VAE and specific techniques to improve the model's understanding of text and speech features, NaturalSpeech can generate speech indistinguishable from humans.
• Conclusion: The paper argues that NaturalSpeech achieves human-level speech quality based on statistical measures (MOS and CMOS) in human evaluations.
• Critical Observations: The evaluation relies on subjective human ratings, which might be influenced by factors beyond speech quality.The research focuses on a single benchmark dataset, limiting generalizability.The paper doesn't explore how NaturalSpeech performs on diverse speaking styles or accents.

• Relevance: This is related to my study because it provides a definition of human-level quality, and this particular model has achieved the highest Mean Opinion Score (MOS) recorded thus far. Hence, I am considering using this model as a basis for my study.

Noufi, C., May, L., & Berger, J. (2023). Context, Perception, Production: A Model of Vocal Persona. PsyArXiv. July, 28.

• Summary: This article introduces a contextualized production-perception model of vocal persona, developed through qualitative analysis of interviews with voice and performance experts. It emphasizes the influence of context on an individual's vocal expression, reflecting the intricacies of human communication.
• RQ: What is the relationship between context, vocal expression, and identity?
• Hypothesis: It is a qualititative study and does not hve a formulated hypothesis. Instead of attempting to falsify a hypothesis as in most quantitative studies, it explores answers to the research question through thematic analysis.
• Conclusion: Speakers actively select different vocal personas and adjust relevant vocal expressions in response to the surrounding context, facilitating a transition in the perception of persona.
• Critical observations: The proposal of the vocal persona model and general conclusions are based on interviews with 21 voice and performance experts, which may have limitations in terms of subjective bias and generalizability beyond this specific context.
• Relevance: This study underscores the necessity for speakers to adapt their speaking styles to accommodate different social contexts, highlighting the significance of context in vocal expression. It proposes the incorporation of vocal persona into expressive vocal synthesis with a three-spoke model and a framework for persona-guided vocalization, enriching the framework of TTS naturalness and expressiveness.

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

APA Citation of an article

• Summary:
• RQ:
• Hypothesis:
• Conclusion:
• Critical observations:
• Relevance:

Synthesis

Synthesis: Conclude with a section that synthesizes the key findings across the articles, highlighting any emerging trends, debates, or future research directions.

Contributors

Contributors: A list of contributors by contribution

• Article Huang et al. 2023: Yilan Wei
• Article 'Context, Perception, Production: A Model of Vocal Persona': Chenyi Lin
• Article NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality: Yi Lei
• Introduction: Chenyi Lin
• Synthesis: All

Theme: ASR

Introduction

The rapid evolution of Automatic Speech Recognition (ASR) technology has been a cornerstone in advancing how humans interact with machines, propelling us towards more seamless and intuitive communication avenues. The focus on ASR technology underscores its pivotal role across a myriad of applications, from enhancing accessibility and providing robust customer support solutions to creating immersive interactive entertainment experiences. Among the most intriguing challenges in this domain is the recognition and interpretation of complex human sentiments such as sarcasm and humor. These nuanced forms of expression, deeply embedded in human language, present unique challenges for ASR systems due to their reliance on contextual cues, background knowledge, and the subtle modulations in tone that conventional speech recognition systems often miss. Our exploration is driven by the imperative to bridge this gap, aiming to refine ASR technology's ability to discern and process these complex sentiments.

Article summaries

• Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

Wang S, Yang C H H, Wu J, et al. Can whisper perform speech-based in-context learning[J]. arXiv preprint arXiv:2309.07081, 2023.

• Summary: The study investigates Whisper ASR models' in-context learning capabilities and proposes a novel SICL method for test-time adaptation without gradient descent, achieving significant WER reductions.
• RQ: The research explores whether Whisper models can perform speech-based in-context learning and how to leverage in-context examples for test-time adaptation efficiently.
• Hypothesis: The hypothesis is that Whisper models can adapt at test time using SICL with context examples from specific dialects or speakers.
• Conclusion: SICL significantly improves ASR performance for Chinese dialects without gradient descent, with k-NN enhancing SICL's efficiency.
• Critical observations: Correct LID settings and k-NN example selection improve Whisper's inference, with language-level adaptation outperforming speaker-level adaptation.
• Relevance: The study is relevant for understanding and enhancing the application of large pre-trained models in automatic speech recognition and dialect adaptation.

Sungjoo Ahn and Hanseok Ko. “Background Noise Reduction via Dual-Channel Scheme for Speech Recognition in Vehicular Environment.” IEEE Transactions on Consumer Electronics 51, no. 1 (February 2005): 22–27. https://doi.org/10.1109/TCE.2005.1405694.

• Summary: The paper proposes a dual-channel noise reduction method aimed at enhancing speech recognition systems within vehicular environments, characterized by significant noise challenges. The authors argue that existing single-channel methods fall short in effectively improving speech recognition performance due to inherent noise complexities in vehicles. The proposed method leverages a high-pass filter combined with an eigen-decomposition front-end processing technique, tested against real multi-channel vehicular corpus. Experimental results indicated a notable improvement in speech recognition performance using various microphone arrangements, showcasing the superiority of the dual-channel approach over traditional single-channel methods.
• RQ: How can the performance of speech recognition systems in vehicular environments be improved through a dual-channel noise reduction scheme?
• Hypothesis: The paper hypothesizes that employing a dual-channel noise reduction scheme, which integrates a high-pass filter with eigen-decomposition front-end processing, can significantly enhance speech recognition performance in noisy vehicular environments by effectively distinguishing speech from background noise.
• Conclusion: Authors concluded that their dual-channel noise reduction method, especially when augmented with a high-pass filter and enhanced eigen-decomposition processing, substantially improves speech recognition accuracy in vehicular settings. The method outperformed standard single-channel noise reduction approaches and showed considerable promise in overcoming the challenges posed by vehicular background noise, thereby validating the hypothesis.
• Critical observations: The study successfully demonstrates the effectiveness of a dual-channel approach in a challenging noise environment. However, the practical deployment of such systems, including the economic implications and the adaptability across different vehicle models and noise conditions, remains less explored. Additionally, while the study marks a significant improvement over existing methods, the scalability of this approach in terms of computational demand and real-time processing capabilities could benefit from further investigation.
• Relevance: This thesis is relevant to the topics in enhancing speech recognition technology area. The innovative approach of combining a dual-channel noise reduction scheme with a high-pass filter and eigen-decomposition method provides a substantial leap forward in developing more reliable and efficient speech recognition systems.

Zhang, Wangyou, and Yanmin Qian. “Weakly-Supervised Speech Pre-Training: A Case Study on Target Speech Recognition.” arXiv, June 29, 2023. http://arxiv.org/abs/2305.16286.

• Summary: This study introduces a new way to teach computers to understand speech by focusing on one person's voice in a noisy place, like when many people talk at once. This method, called TS-HuBERT, uses extra information about the speaker's voice to improve speech recognition, especially in challenging situations with lots of background noise. Tests showed that TS-HuBERT does a better job than other similar methods, making it a promising approach for better understanding speech in noisy environments.
• RQ: Can we use extra information about who is speaking to help computers better recognize speech in noisy settings?
• Hypothesis: By using additional information about the speaker, the TS-HuBERT method can focus on the target speaker's voice more effectively, even when other voices or noises are present.
• Conclusion: TS-HuBERT improves speech recognition by focusing on the target speaker's voice, outperforming other current methods. This approach is particularly useful for recognizing speech in noisy places where many people are talking at once.
• Critical observations:
  • TS-HuBERT can be adjusted to different speech recognition tasks, showing its versatility.
  • Although it needs extra information about the speaker's voice, this method greatly enhances the computer's ability to focus on and understand the target speaker in noisy situations.
  • There is still room for improvement, especially in very noisy environments, indicating potential areas for future research.
• Relevance: This study is directly relevant to the topic to help computers understand speech better in challenging environments, like when many people are talking at the same time. By focusing on a specific speaker's voice, TS-HuBERT could make speech recognition technology more effective in real-world situations.

Bae, S., Kim, J.-W., Cho, W.-Y., Baek, H., Son, S., Lee, B., Ha, C., Tae, K., Kim, S., & Yun, S.-Y. (2023). Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification. Retrieved from https://arxiv.org/abs/2305.14032v4

Summary: The study introduces a novel approach for respiratory sound classification, leveraging a pretrained Audio Spectrogram Transformer (AST) model, alongside a new Patch-Mix augmentation technique and Patch-Mix Contrastive Learning. These methods are designed to address the challenges of medical data scarcity and enhance model performance on the ICBHI dataset. The approach sets a new state-of-the-art performance benchmark, improving the classification Score by 4.08% over previous methods.

• RQ: Can a pretrained Audio Spectrogram Transformer (AST) model, combined with Patch-Mix augmentation and Patch-Mix Contrastive Learning, effectively improve respiratory sound classification, especially in the context of the ICBHI dataset?
• Hypothesis: The hypothesis posits that leveraging a pretrained AST model, which has been trained on large-scale visual and audio datasets, can be effectively generalized to respiratory sound classification tasks. Additionally, it suggests that the introduction of Patch-Mix augmentation and Patch-Mix Contrastive Learning can further enhance model performance by addressing the scarcity of medical data and the challenges of leveraging such data for deep learning models.
• Conclusion: The study concludes that the proposed approach, combining a pretrained AST model with Patch-Mix augmentation and Patch-Mix Contrastive Learning, significantly enhances respiratory sound classification. This method achieved state-of-the-art performance on the ICBHI dataset, demonstrating the effectiveness of the proposed techniques in improving classification accuracy in the face of limited medical data availability and complex data characteristics.
• Critical observations:
  • Pre-training on both visual and audio domains using the AST model shows substantial improvements in generalizing to respiratory sound classification tasks.
  • The Patch-Mix augmentation technique, which randomly mixes patches between different samples, and the Patch-Mix Contrastive Learning method, which distinguishes mixed representations in the latent space, effectively mitigate the overfitting issue and enhance model robustness.
  • The study's methodology offers a significant performance increase, demonstrating the potential of attention-based models and contrastive learning in medical sound classification.
• Relevance: This research holds relevance to Automatic Speech Recognition (ASR) by showcasing the utility of attention-based models like the AST in capturing long-range dependencies in audio data. The techniques developed for respiratory sound classification, particularly the effective use of pretrained models and innovative augmentation strategies, can inform similar challenges in ASR, including dealing with limited training data and enhancing model generalization across diverse audio inputs.

Gairola, S., Tom, F., Kwatra, N., & Jain, M. (2021). RESPIRENET: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting. Retrieved from https://arxiv.org/abs/2011.00196v2

• Summary: The study introduces RespireNet, a CNN-based model for classifying respiratory sounds, particularly focusing on addressing the challenge posed by the small size of the largest available respiratory dataset, ICBHI, which consists of only 6,898 breathing cycles. The study proposes a suite of novel techniques including device-specific fine-tuning, concatenation-based augmentation, blank region clipping, and smart padding to efficiently utilize this small dataset. Extensive evaluation on the ICBHI dataset demonstrates significant improvements over state-of-the-art results for 4-class classification by 2.2%.
• RQ: Can a simple CNN-based model, when combined with specific data utilization techniques, accurately classify respiratory sounds from a limited-sized dataset, overcoming the challenges of data scarcity and variability?
• Hypothesis: The study hypothesizes that even with a small dataset, a simple network architecture, if supplemented with innovative techniques for data utilization and augmentation, can accurately classify respiratory sounds. These techniques include addressing dataset characteristics such as device variability, class imbalance, and varying audio lengths that traditionally inhibit effective DNN training.
• Conclusion: RespireNet, along with the proposed data utilization techniques, significantly improves the accuracy of respiratory sound classification, achieving new state-of-the-art performance on the ICBHI dataset for both 2-class and 4-class classification tasks. The study concludes that focusing on efficient data utilization and addressing specific dataset characteristics can compensate for the limitations posed by small-sized datasets.
• Critical observations:
  1. Transfer learning from pre-trained ImageNet models proves beneficial, suggesting that even unrelated domain knowledge can improve model performance.
  2. Concatenation-based augmentation effectively addresses class imbalance, significantly improving classification of underrepresented classes.
  3. Device-specific fine-tuning is essential for generalizing across different recording devices, highlighting the impact of hardware variability on model performance.
  4. Techniques like smart padding and blank region clipping are crucial for dealing with variable-length audio samples and irrelevant frequency regions, respectively, ensuring the model focuses on relevant features.
• Relevance: The challenges and solutions presented in this study have direct implications for ASR, especially in scenarios where data is scarce or highly variable. Techniques such as smart data augmentation, device-specific adjustments, and focusing on relevant audio features can be applied to improve ASR systems' robustness and accuracy in diverse conditions. Furthermore, the emphasis on efficient data utilization and simple model architectures can inspire similar approaches in ASR research to overcome data-related limitations.

Yang, R., Lv, K., Huang, Y., Sun, M., Li, J., & Yang, J. (2023). Respiratory Sound Classification by Applying Deep Neural Network with a Blocking Variable. Applied Sciences, 13(6956). https://doi.org/10.3390/app13126956

• Summary: The paper introduces a deep neural network named Blnet for classifying respiratory sounds, incorporating features from ResNet, GoogleNet, and self-attention mechanisms to tackle the non-IID (not independently and identically distributed) data problem and imbalanced data issues. The model demonstrated improved performance on the ICBHI 2017 respiratory sound database, showcasing a significant advancement in sensitivity and specificity rates over existing methods.
• RQ: How can a deep neural network be optimized for classifying respiratory sounds to facilitate the early detection of respiratory diseases, considering challenges such as non-IID data and imbalanced datasets?
• Hypothesis: The integration of ResNet, GoogleNet, and self-attention mechanisms into a deep neural network, alongside a two-stage training process and mix-up data augmentation within clusters, can significantly improve the classification accuracy of respiratory sounds, even with imbalanced and non-IID data challenges.
• Conclusion: The Blnet model successfully addressed the challenges of non-IID and imbalanced datasets in respiratory sound classification, achieving a 4.22% improvement in average score and a 12.61% improvement in sensitivity over state-of-the-art results. This performance enhancement underscores the efficacy of the proposed network architecture and training strategies.
• Critical observations:
  • The two-stage training process and the introduction of a blocking variable proved effective in managing non-IID data, suggesting the importance of considering data distribution in deep learning models.
  • Mix-up data augmentation within clusters and the use of multiple input transformations (STFT and WT) were critical in addressing data imbalance and enhancing model robustness.
  • The self-attention mechanism played a key role in capturing global dependencies within the data, improving the model's feature extraction capabilities.
  • Simplifying the loss function to handle a four-class classification task as two independent binary classification tasks was found to enhance training effectiveness.
• Relevance: The techniques and findings of this study have direct implications for ASR systems, particularly in enhancing model performance with non-IID and imbalanced datasets. The methods for improving feature extraction and classification in the context of respiratory sound analysis can inform approaches to noise reduction, signal processing, and robust model training in ASR technologies. Furthermore, the attention mechanisms and data augmentation strategies could be adapted to improve ASR systems' ability to deal with diverse and challenging acoustic environments.

Zhou, Rui, Xian Li, Ying Fang, and Xiaofei Li. “Mel-FullSubNet: Mel-Spectrogram Enhancement for Improving Both Speech Quality and ASR.” arXiv, February 21, 2024. http://arxiv.org/abs/2402.13511.

• Summary: This paper introduces Mel-FullSubNet, a network designed for enhancing speech quality and automatic speech recognition (ASR) performance. It focuses on improving both the clarity of speech and its recognizability by machines in noisy conditions. The technique enhances Mel-spectrograms of speech, which can then be used directly for ASR or converted back into speech waveforms using a neural vocoder. The method combines full-band and sub-band network processing, proving to be more effective for ASR and speech quality enhancement compared to previous approaches.
• RQ: Can Mel-spectrogram enhancement via Mel-FullSubNet improve both speech quality and automatic speech recognition performance in noisy conditions?
• Hypothesis: By enhancing Mel-spectrograms using the Mel-FullSubNet, which combines full-band and sub-band processing, both speech quality and ASR performance can be significantly improved in noisy environments.
• Conclusion: Mel-FullSubNet successfully enhances speech quality and ASR performance, outperforming several existing methods. It shows particular strength in providing cleaner speech signals and more accurate ASR results by focusing on Mel-spectrogram enhancement and efficiently leveraging neural vocoders for waveform generation.
• Critical observations:
  • Mel-FullSubNet demonstrates superior generalization to unseen data and environments, a critical advantage for real-world applications.
  • The method's efficacy is underscored by its performance on various datasets, indicating its robustness and adaptability.
  • While Mel-FullSubNet requires more computational resources due to its neural vocoder component, its efficiency and output quality justify the additional cost.
• Relevance:This study is directly relevant to the topic to the challenge of enhancing speech recognition systems in noisy conditions, a common problem in real-world applications. By focusing on Mel-spectrogram enhancement, Mel-FullSubNet provides a novel approach that benefits both speech clarity and ASR accuracy, making it a valuable reference for further research in speech processing technology.

Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., & Poria, S. (2019). Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). arXiv:1906.01815v1.

Summary: The paper introduces a novel approach to sarcasm detection by leveraging multimodal data. Recognizing that sarcasm often involves incongruities not just in text but also in vocal tone and facial expressions, the authors propose the first dataset, MUStARD, for sarcasm detection using audio-visual cues alongside textual data. This dataset, compiled from popular TV shows, is annotated for sarcasm, aiming to facilitate the development of models that can better understand sarcasm through the integration of multiple modes of communication.

RQ: How can incorporating multimodal cues (textual, audio, and visual) improve the automatic classification of sarcasm compared to relying on textual data alone?

Hypothesis: The paper hypothesizes that the inclusion of multimodal information (audio and visual cues, along with textual data) can significantly enhance the performance of sarcasm detection models, reducing the relative error rate by up to 12.9% in F-score when compared to models that use only individual modalities.

Conclusion: The research demonstrates that multimodal models significantly outperform unimodal variants in sarcasm detection, with a notable reduction in error rate. The findings underscore the importance of considering multiple communication cues, beyond just text, for effectively identifying sarcasm. The MUStARD dataset is also introduced as a valuable resource for future research in multimodal sarcasm detection.

Critical Observations:

1. Sarcasm detection benefits from multimodal analysis, including textual, audio, and visual data, highlighting the complex nature of sarcasm as a communicative act that often relies on the interplay of various signals.
2. The MUStARD dataset fills a critical gap in research resources, providing a foundation for exploring how different modalities contribute to the understanding of sarcasm.
3. The study's methodology, focusing on a balanced dataset and robust multimodal feature extraction techniques, sets a precedent for future work in this area.

Relevance: This research is highly relevant to the field of natural language processing and computational linguistics, particularly in the area of sentiment analysis and affective computing. It pushes the boundaries of sarcasm detection by moving beyond text analysis to include audio and visual cues, offering insights into more holistic approaches to understanding human communication. The findings and the MUStARD dataset can significantly impact the development of more nuanced and effective computational models for detecting sarcasm and other complex emotional or figurative language use cases.

Zhang, Yazhou, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. “CMMA: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations,” n.d.

• Summary: This study introduces the CMMA dataset for benchmarking multi-affection detection in Chinese multi-modal conversations, focusing on sentiment, emotion, sarcasm, and humor. The dataset comprises annotations from a variety of TV series to reflect diverse affective expressions and supports both single-task and multi-task learning paradigms for affective computing research.
• RQ: How multi-modal cues and conversational context influence the detection of multiple affects, including sentiment, emotion, sarcasm, and humor, in Chinese multi-party conversations?
• Hypothesis: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations" likely centers on the premise that incorporating multi-modal data (text, video, audio) and conversational context significantly improves the accuracy and effectiveness of detecting multiple affects (sentiment, emotion, sarcasm, humor) in multi-party conversations. The study posits that the interplay between different modalities and the contextual understanding of conversations enhances the model's ability to interpret complex human affective expressions.
• Conclusion: The findings demonstrate that conversational context and multi-modal data significantly enhance affect detection tasks. The study also highlights the importance of multi-affect annotation for understanding complex human communications, suggesting the CMMA dataset as a valuable resource for future affective computing research.
• Critical observations: While the dataset offers comprehensive insights into multi-affect detection, its focus on Chinese TV series may limit its applicability across different linguistic and cultural contexts. Additionally, the inherent subjectivity of affect annotation poses challenges to achieving unbiased affect detection.
• Relevance: This study is pertinent to my thesis as it provides an opportunity to delve into how various feature fusion methods impact the accuracy of sarcasm recognition in Mandarin using multimodal data. Additionally, the CMMA dataset is highly beneficial to my research because it is among the few Chinese datasets that include labels for sarcasm, offering a valuable resource for studying sarcasm recognition within Mandarin-specific contexts using multimodal information.

Synthesis

The articles reviewed collectively contribute to the ASR field, showing a trend towards multimodal data use, context awareness, and noise reduction techniques to address complexities in human speech such as sarcasm and humor. Key observations include the importance of integrating audio, visual, and textual data for better sarcasm detection, the effectiveness of dual-channel noise reduction in vehicular environments, and the application of deep learning for respiratory sound classification and speech enhancement in noisy settings. Challenges mentioned across these studies involve data scarcity, handling diverse dialects, and computational demands. Future research directions suggest a focus on improving ASR systems' adaptability across languages and cultures, better managing non-IID and imbalanced data, and enhancing emotional intelligence in speech recognition. These findings indicate ongoing efforts to make ASR technologies more intuitive and effective in complex human-machine interactions.

Contributors

Contributors: A list of contributors by contribution

• Article Sungjoo Ahn and Hanseok Ko (2005): Dongwen Zhu
• Article Zhang and Qian (2023): Dongwen Zhu
• Article Zhou et al. (2024): Dongwe Zhu
• Article Bae et al. (2023): Soogyeong Shin
• Article Gairola et al. (2021): Soogyeong Shin
• Article Yang et al. (2023): Soogyeong Shin
• Article Wang et al. (2023): Yaling Deng
• Article Castro et al. (2019) : Erin Shi
• Article Zhang et al. (2021): Youyang Cai
• Introduction: All
• Synthesis: All

Miscellaneous

This last section corresponds to articles that did not fit well inside other themes.

Synthetically improving foreign-accented speech recognition

Introduction

More often than not, speech corpora either contain only native speech, or the non-native subset is significantly underrepresented. At the same time, gender and foreign accent are the most salient factors contributing to changes in the acoustics of speech. However, not only are there numerous possible combinations of L1 and L2s, but the annotation and labelling os recordings to a suitable degree (e.g. age of L2 acquisition, country of origin, L1, L2 proficiency, language of education etc. are all factors that should be reported in order to make the speech resources reliable and usable) are laborious and expensive.

In light of these challenges, methods of synthetical data augmentation have been recently explored in the literature. While creating synthetically-accented data through accent conversion models (ACMs) is a straightforward, inexpensive, and oof-the-shelf approach, it is not without limitations and the degree to which recognition performance is improved through such approaches depends on several factors. The following three articles provide some insight into these approaches and highlight both major advantages and persistent challenges.

Zhao et al. (2018): Accent conversion using phonetic posteriograms

Summary: Accent conversion (AC) means transforming non-native speech to sound as if the speaker had a native accent, or vice-versa. The main challenge faced in traditional methods of voice conversion is decoupling the speaker’s voice quality from their pronunciation (i.e. teasing apart accent information and keeping everything else acoustically unchanged). Additionally, when mapping source spectra from a native speaker into the acoustic space of an L2 speaker, previous attempts focus on acoustic similarity: changing formants- and pitch trajectories, blending spectral envelopes. The alternative used here is, in turn, is phonetic similarity, which maps source to target based on an intermediate phonetic label. The phonetic posteriograms are computed using a DNN-based acoustic model. The distance between these phonetic posterior feature vectors is calculated to find the closest pairs of frames between source (native) and target (L2) speakers. The frame pairs are used to train a GMM. The two baselines used are acoustic similarity matching and dynamic time warping.

Experimental setup: get Kaldi DNN acoustic model, train it on Librispeech data, get native English speech (CMUArctic) and non-native recordings (Hindi, Korean, Arabic), use STRAIGHT for speech decomposition, MFCC extraction, train GMMs (128 components), synthesize speech by reconstructing spectrograms and adding aperiodicity.

RQ: How can accent-related features be successfully decoupled from speaker-related features, to achieve non-native to native voice conversion while preserving speech quality?

Results: Synthesized results were compared to baselines through listening tasks using Mechanical Turk (rating acoustic quality, speaker identity y/n, nativeness of resynthesized speech):

• significantly higher acoustic quality ratings compared to baselines.
• comparable speaker identity scores.
• strong preferrence for posteriogram conversions by native EN speakers as more ‘native-like‘ compared to baselines and original L2 utterances.

Critical observations: This paper addressed the opposite issue, namely converting foreign-accented speech to sound like native one (mainly for educational purposes). This still means you need to figure out which features are related to accent, and which features are related to anything else, but is arguably the easier thing to do, as it requires to drop information instead of successfully adding it. Additionally, the approach is not entirely explainable, because posteriograms are encoder features and it's not always transparent what is learned to be most relevant. Lastly, this approach likely works increasingly worse the fewer speakers there are in a dataset. Even if you accented speech data, one speaker can only have one accent, so in case the number of speakers is small, the model might learn to encode speaker identity instead of accent features.

Relevance: It is important to know that given enough speakers and enough data, accent features can be decoupled from other speech features and dropeed to obtain a higher perceived 'nativeness' of the speech.

Jin et al. (2023): Voice-preserving zero-shot multiple accent conversion

Summary: Separating accent from speaker identity is usually the hardest, because each speaker in the dataset has one single accent. Previous attempts at doing this include:

• use adversarial learning to get a discriminator to wipe out speaker-dependent information from content embeddings.
• quantization of different features in speech to obscure undesired information.

The main problem with conventional approaches to conversion is that they very often require available utterances with the same text in both source and target accent, making their applicability very limited. Alternatively, different approaches require either training or fine-tuning on the input utterances.

The current paper uses a pronunciation encoder, an acoustic encoder, and a HiFiGAN voice decoder. During training, the model minimises reconstruction loss between input and output mel-spectrograms. The pronunciation encoder synthesizes accent-dependent pronunciation sequences using accent IDs. The acoustic encoder mapss MFCCs and periodicity features to a single vector, while adversarial training removes accent information. Lastly, the decoder reconstructs waveforms from the processed features. The model is evaluated on audio quality, speaker similarity, and accent conversion effectiveness.

Results: Results indicate it maintains comparable audio quality to the original, maintains speaker similarity, and is efficient in replicating perceived nativeneess. However, listeners struggled to identify synthesized accents if they were unfamiliar with the target language (e.g. a native US listener could not classify a Korean accent on English as such, but a bilingual Korean-American listener could). Overall, the paper presents one of the best performing ACMs, that is able to preserve both speaker identity and acoustic quality during conversion.

Critical observations: I think this paper achives a lot given that it's zero shot, but I am a bit critical about just how 'zero-shot' it truly is. They use a pre-trained acoustic model and while they do not require accent labels or speaker IDs, it seems that their training set contains over 24h of accented speech for all accents that they're synthesizing in. Additionally, none of their code is openly available, which is understandable for a private corporation like Meta, but it's still a bit disappointing.

Klumpp et al. (2023): Synthetic cross-accent data augmentation for ASR

Summary: Foreign-accentes speech is usually underrepresented in, if not absent from speech corpora. Auxiliary input (learned accent embeddings, intermediate wav2vec2.0 representations) can address the decreased ASR recognition on this type of speech; the challenge remains that of achieving good accent conversion while preserving source speaker voice characteristics. The current approach builds on a pre-existing ACM by Jin et al. (2023) -- see above -- and aims to provide synthetic ASR training data using it. Phonetic knowledge is crucially injected into training to improve accent-specific pronunciation, and learnable accent representations are introduced to allow for variable accent strengths and adaptability to unseen accents.

The experimental setup involved evaluating two ASR models using Librispeech data. The first model (Base) utilized an efficient memory transformer followed by a recurrent neural transducer (RNNT), while the second model (HuBERT) had a similar structure with adjustments in channel configurations and dropout probabilities. The ASR models were tested on Librispeech data and accents from L2-Arctic corpus and Accented Vox Populi (AVP) dataset.

In experiments, the baseline ASR systems were trained without synthetic accented speech data, then evaluated. Three additional ASR models were trained with a combination of real and synthetic accented data, using a ratio of 80% real and 20% synthetic data. The ratio remained consistent across all accents. Finally, learned accent embeddings from L2-Arctic samples were visualized using t-SNE plots to assess their suitability for encoding accent information in an Accent Conversion Model (ACM).

RQ: Is it possible to improve ASR of accented speech with synthetic samples of a particular accent?

Results: The inclusion of one synthetic accent during ASR training had a positive effect on recognition results for that particular accent, which was a clear indicator that the ACM was able to synthesize a sufficient degree of accentedness. At the same time, HuBERT'd performance decreased with the use of synthetic data, likely due to the fact that it was not pre-trained on any and fine-tuning did not do enough. The Base model, which was trained from scratch, had a much grater benefit from the synthetic data. Notably, even when all seven accents were introduced in training, this did not improve performance on other unseen accents.

Overall, including one synthetic accent improved performance on that accent; and including several accents improved performance on those accents, but none of the conditions improved recognition on accents not seen in training. Additionally, pre-trained HuBERT did not benefit much from additional synthetic data fine-tuning, whereas a model trained from scratch saw much greater benefit from this approach.

Critical observations: Again, none of this replicable because the code is not available. It would have been also interesting to see a bit more ASR models be tested on this; this particular comparison does highlight the pre-trained/trained from scratch distinction in performance on this task, but there are other models that are seemingly good candidates and were not included.

Relevance: The authors show the potential for using synthetically accented data as a data augmentation approach to improve ASR performance on foreign-accented speech.

General insights

The synthesis of accented speech as a data augmentation method in ASR is promising for improving recognition performance on non-native speech. The three articles reviewed provide valuable insights into accent conversion methods and their implications for ASR systems. Zhao et al. (2018) shows the effectiveness of phonetic posteriograms in converting foreign-accented speech to sound more native-like and successfully decouples accent-related features from other speech characteristics. Jin et al. (2023) proposed a zero-shot multiple accent conversion approach, maintaining audio quality and speaker identity during conversion, albeit with limitations in accent classification for unfamiliar listeners. Klumpp et al. (2023) extended this work by integrating synthetic accented speech data into ASR training, showing improvements in recognition performance on the trained accents. However, the effectiveness varied depending on the model architecture, with pre-trained models benefiting less from synthetic data than models trained from scratch. Despite promising results, the lack of code availability and limited generalizability to unseen accents pose challenges for broader adoption. Overall, while accent conversion models offer a promising strategy for data augmentation in ASR, further research should focus on generalization and replicability for real-world applications.

References

Jin, M., Serai, P., Wu, J., Tjandra, A., Manohar, V., & He, Q. (2023, June). Voice-preserving zero-shot multiple accent conversion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

Klumpp, P., Chitkara, P., Sarı, L., Serai, P., Wu, J., Veliche, I. E., ... & He, Q. (2023). Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition. arXiv preprint arXiv:2303.00802.

Zhao, G., Sonsaat, S., Levis, J., Chukharev-Hudilainen, E., & Gutierrez-Osuna, R. (2018, April). Accent conversion using phonetic posteriorgrams. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5314-5318). IEEE.

Contributors:

The section Synthetically improving foreign-accented speech recognition was written by Maria Tepei.