Editing State-of-the-art (section)

=== Article summaries ===

* Article summaries and analyses: Each article receives a subsection including a summary (reference to RQ and hypothesis), critical analysis, and discuss its relevance to your theme.

==== Wang S, Yang C H H, Wu J, et al. Can whisper perform speech-based in-context learning[J]. arXiv preprint arXiv:2309.07081, 2023. ====

* Summary: The study investigates Whisper ASR models' in-context learning capabilities and proposes a novel SICL method for test-time adaptation without gradient descent, achieving significant WER reductions.
* RQ: The research explores whether Whisper models can perform speech-based in-context learning and how to leverage in-context examples for test-time adaptation efficiently.
* Hypothesis: The hypothesis is that Whisper models can adapt at test time using SICL with context examples from specific dialects or speakers.
* Conclusion: SICL significantly improves ASR performance for Chinese dialects without gradient descent, with k-NN enhancing SICL's efficiency.
* Critical observations: Correct LID settings and k-NN example selection improve Whisper's inference, with language-level adaptation outperforming speaker-level adaptation.
* Relevance: The study is relevant for understanding and enhancing the application of large pre-trained models in automatic speech recognition and dialect adaptation.

==== Sungjoo Ahn and Hanseok Ko. “Background Noise Reduction via Dual-Channel Scheme for Speech Recognition in Vehicular Environment.” IEEE Transactions on Consumer Electronics 51, no. 1 (February 2005): 22–27. <nowiki>https://doi.org/10.1109/TCE.2005.1405694</nowiki>. ====

* Summary: The paper proposes a dual-channel noise reduction method aimed at enhancing speech recognition systems within vehicular environments, characterized by significant noise challenges. The authors argue that existing single-channel methods fall short in effectively improving speech recognition performance due to inherent noise complexities in vehicles. The proposed method leverages a high-pass filter combined with an eigen-decomposition front-end processing technique, tested against real multi-channel vehicular corpus. Experimental results indicated a notable improvement in speech recognition performance using various microphone arrangements, showcasing the superiority of the dual-channel approach over traditional single-channel methods.
* RQ: How can the performance of speech recognition systems in vehicular environments be improved through a dual-channel noise reduction scheme?
* Hypothesis: The paper hypothesizes that employing a dual-channel noise reduction scheme, which integrates a high-pass filter with eigen-decomposition front-end processing, can significantly enhance speech recognition performance in noisy vehicular environments by effectively distinguishing speech from background noise.
* Conclusion: Authors concluded that their dual-channel noise reduction method, especially when augmented with a high-pass filter and enhanced eigen-decomposition processing, substantially improves speech recognition accuracy in vehicular settings. The method outperformed standard single-channel noise reduction approaches and showed considerable promise in overcoming the challenges posed by vehicular background noise, thereby validating the hypothesis.
* Critical observations: The study successfully demonstrates the effectiveness of a dual-channel approach in a challenging noise environment. However, the practical deployment of such systems, including the economic implications and the adaptability across different vehicle models and noise conditions, remains less explored. Additionally, while the study marks a significant improvement over existing methods, the scalability of this approach in terms of computational demand and real-time processing capabilities could benefit from further investigation.
* Relevance: This thesis is relevant to the topics in enhancing speech recognition technology area. The innovative approach of combining a dual-channel noise reduction scheme with a high-pass filter and eigen-decomposition method provides a substantial leap forward in developing more reliable and efficient speech recognition systems.

==== Zhang, Wangyou, and Yanmin Qian. “Weakly-Supervised Speech Pre-Training: A Case Study on Target Speech Recognition.” arXiv, June 29, 2023. <nowiki>http://arxiv.org/abs/2305.16286</nowiki>. ====
* Summary: This study introduces a new way to teach computers to understand speech by focusing on one person's voice in a noisy place, like when many people talk at once. This method, called TS-HuBERT, uses extra information about the speaker's voice to improve speech recognition, especially in challenging situations with lots of background noise. Tests showed that TS-HuBERT does a better job than other similar methods, making it a promising approach for better understanding speech in noisy environments.
* RQ: Can we use extra information about who is speaking to help computers better recognize speech in noisy settings?
* Hypothesis: By using additional information about the speaker, the TS-HuBERT method can focus on the target speaker's voice more effectively, even when other voices or noises are present.
* Conclusion: TS-HuBERT improves speech recognition by focusing on the target speaker's voice, outperforming other current methods. This approach is particularly useful for recognizing speech in noisy places where many people are talking at once.
* Critical observations: 
** TS-HuBERT can be adjusted to different speech recognition tasks, showing its versatility.
** Although it needs extra information about the speaker's voice, this method greatly enhances the computer's ability to focus on and understand the target speaker in noisy situations.
** There is still room for improvement, especially in very noisy environments, indicating potential areas for future research.
* Relevance: This study is directly relevant to the topic to help computers understand speech better in challenging environments, like when many people are talking at the same time. By focusing on a specific speaker's voice, TS-HuBERT could make speech recognition technology more effective in real-world situations.

==== Bae, S., Kim, J.-W., Cho, W.-Y., Baek, H., Son, S., Lee, B., Ha, C., Tae, K., Kim, S., & Yun, S.-Y. (2023). Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification. Retrieved from <nowiki>https://arxiv.org/abs/2305.14032v4</nowiki> ====
Summary: The study introduces a novel approach for respiratory sound classification, leveraging a pretrained Audio Spectrogram Transformer (AST) model, alongside a new Patch-Mix augmentation technique and Patch-Mix Contrastive Learning. These methods are designed to address the challenges of medical data scarcity and enhance model performance on the ICBHI dataset. The approach sets a new state-of-the-art performance benchmark, improving the classification Score by 4.08% over previous methods.
*RQ: Can a pretrained Audio Spectrogram Transformer (AST) model, combined with Patch-Mix augmentation and Patch-Mix Contrastive Learning, effectively improve respiratory sound classification, especially in the context of the ICBHI dataset?
* Hypothesis: The hypothesis posits that leveraging a pretrained AST model, which has been trained on large-scale visual and audio datasets, can be effectively generalized to respiratory sound classification tasks. Additionally, it suggests that the introduction of Patch-Mix augmentation and Patch-Mix Contrastive Learning can further enhance model performance by addressing the scarcity of medical data and the challenges of leveraging such data for deep learning models.
* Conclusion: The study concludes that the proposed approach, combining a pretrained AST model with Patch-Mix augmentation and Patch-Mix Contrastive Learning, significantly enhances respiratory sound classification. This method achieved state-of-the-art performance on the ICBHI dataset, demonstrating the effectiveness of the proposed techniques in improving classification accuracy in the face of limited medical data availability and complex data characteristics.
* Critical observations:
** Pre-training on both visual and audio domains using the AST model shows substantial improvements in generalizing to respiratory sound classification tasks.
** The Patch-Mix augmentation technique, which randomly mixes patches between different samples, and the Patch-Mix Contrastive Learning method, which distinguishes mixed representations in the latent space, effectively mitigate the overfitting issue and enhance model robustness.
** The study's methodology offers a significant performance increase, demonstrating the potential of attention-based models and contrastive learning in medical sound classification.
* Relevance: This research holds relevance to Automatic Speech Recognition (ASR) by showcasing the utility of attention-based models like the AST in capturing long-range dependencies in audio data. The techniques developed for respiratory sound classification, particularly the effective use of pretrained models and innovative augmentation strategies, can inform similar challenges in ASR, including dealing with limited training data and enhancing model generalization across diverse audio inputs.

==== Gairola1, S., Tom, F., Kwatra1, N., & Jain1, M. (2021). RESPIRENET: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting. Retrieved from <nowiki>https://arxiv.org/abs/2011.00196v2</nowiki> ====
*Summary: The study introduces RespireNet, a CNN-based model for classifying respiratory sounds, particularly focusing on addressing the challenge posed by the small size of the largest available respiratory dataset, ICBHI, which consists of only 6,898 breathing cycles. The study proposes a suite of novel techniques including device-specific fine-tuning, concatenation-based augmentation, blank region clipping, and smart padding to efficiently utilize this small dataset. Extensive evaluation on the ICBHI dataset demonstrates significant improvements over state-of-the-art results for 4-class classification by 2.2%.
* RQ: Can a simple CNN-based model, when combined with specific data utilization techniques, accurately classify respiratory sounds from a limited-sized dataset, overcoming the challenges of data scarcity and variability?
* Hypothesis: The study hypothesizes that even with a small dataset, a simple network architecture, if supplemented with innovative techniques for data utilization and augmentation, can accurately classify respiratory sounds. These techniques include addressing dataset characteristics such as device variability, class imbalance, and varying audio lengths that traditionally inhibit effective DNN training.
* Conclusion: RespireNet, along with the proposed data utilization techniques, significantly improves the accuracy of respiratory sound classification, achieving new state-of-the-art performance on the ICBHI dataset for both 2-class and 4-class classification tasks. The study concludes that focusing on efficient data utilization and addressing specific dataset characteristics can compensate for the limitations posed by small-sized datasets.
* Critical observations:
*# Transfer learning from pre-trained ImageNet models proves beneficial, suggesting that even unrelated domain knowledge can improve model performance.
*# Concatenation-based augmentation effectively addresses class imbalance, significantly improving classification of underrepresented classes.
*# Device-specific fine-tuning is essential for generalizing across different recording devices, highlighting the impact of hardware variability on model performance.
*# Techniques like smart padding and blank region clipping are crucial for dealing with variable-length audio samples and irrelevant frequency regions, respectively, ensuring the model focuses on relevant features.
* Relevance: The challenges and solutions presented in this study have direct implications for ASR, especially in scenarios where data is scarce or highly variable. Techniques such as smart data augmentation, device-specific adjustments, and focusing on relevant audio features can be applied to improve ASR systems' robustness and accuracy in diverse conditions. Furthermore, the emphasis on efficient data utilization and simple model architectures can inspire similar approaches in ASR research to overcome data-related limitations.

==== Yang, R., Lv, K., Huang, Y., Sun, M., Li, J., & Yang, J. (2023). Respiratory Sound Classification by Applying Deep Neural Network with a Blocking Variable. ''Applied Sciences'', 13(6956). <nowiki>https://doi.org/10.3390/app13126956</nowiki> ====
*Summary: The paper introduces a deep neural network named Blnet for classifying respiratory sounds, incorporating features from ResNet, GoogleNet, and self-attention mechanisms to tackle the non-IID (not independently and identically distributed) data problem and imbalanced data issues. The model demonstrated improved performance on the ICBHI 2017 respiratory sound database, showcasing a significant advancement in sensitivity and specificity rates over existing methods.
* RQ: How can a deep neural network be optimized for classifying respiratory sounds to facilitate the early detection of respiratory diseases, considering challenges such as non-IID data and imbalanced datasets?
* Hypothesis: The integration of ResNet, GoogleNet, and self-attention mechanisms into a deep neural network, alongside a two-stage training process and mix-up data augmentation within clusters, can significantly improve the classification accuracy of respiratory sounds, even with imbalanced and non-IID data challenges.
* Conclusion: The Blnet model successfully addressed the challenges of non-IID and imbalanced datasets in respiratory sound classification, achieving a 4.22% improvement in average score and a 12.61% improvement in sensitivity over state-of-the-art results. This performance enhancement underscores the efficacy of the proposed network architecture and training strategies.
* Critical observations:
** The two-stage training process and the introduction of a blocking variable proved effective in managing non-IID data, suggesting the importance of considering data distribution in deep learning models.
** Mix-up data augmentation within clusters and the use of multiple input transformations (STFT and WT) were critical in addressing data imbalance and enhancing model robustness.
** The self-attention mechanism played a key role in capturing global dependencies within the data, improving the model's feature extraction capabilities.
** Simplifying the loss function to handle a four-class classification task as two independent binary classification tasks was found to enhance training effectiveness.
* Relevance: The techniques and findings of this study have direct implications for ASR systems, particularly in enhancing model performance with non-IID and imbalanced datasets. The methods for improving feature extraction and classification in the context of respiratory sound analysis can inform approaches to noise reduction, signal processing, and robust model training in ASR technologies. Furthermore, the attention mechanisms and data augmentation strategies could be adapted to improve ASR systems' ability to deal with diverse and challenging acoustic environments.

==== Zhou, Rui, Xian Li, Ying Fang, and Xiaofei Li. “Mel-FullSubNet: Mel-Spectrogram Enhancement for Improving Both Speech Quality and ASR.” arXiv, February 21, 2024. <nowiki>http://arxiv.org/abs/2402.13511</nowiki>. ====

*Summary: This paper introduces Mel-FullSubNet, a network designed for enhancing speech quality and automatic speech recognition (ASR) performance. It focuses on improving both the clarity of speech and its recognizability by machines in noisy conditions. The technique enhances Mel-spectrograms of speech, which can then be used directly for ASR or converted back into speech waveforms using a neural vocoder. The method combines full-band and sub-band network processing, proving to be more effective for ASR and speech quality enhancement compared to previous approaches.
* RQ: Can Mel-spectrogram enhancement via Mel-FullSubNet improve both speech quality and automatic speech recognition performance in noisy conditions?
* Hypothesis: By enhancing Mel-spectrograms using the Mel-FullSubNet, which combines full-band and sub-band processing, both speech quality and ASR performance can be significantly improved in noisy environments.
* Conclusion: Mel-FullSubNet successfully enhances speech quality and ASR performance, outperforming several existing methods. It shows particular strength in providing cleaner speech signals and more accurate ASR results by focusing on Mel-spectrogram enhancement and efficiently leveraging neural vocoders for waveform generation.
* Critical observations: 
** Mel-FullSubNet demonstrates superior generalization to unseen data and environments, a critical advantage for real-world applications.
** The method's efficacy is underscored by its performance on various datasets, indicating its robustness and adaptability.
** While Mel-FullSubNet requires more computational resources due to its neural vocoder component, its efficiency and output quality justify the additional cost.
* Relevance:This study is directly relevant to the topic to the challenge of enhancing speech recognition systems in noisy conditions, a common problem in real-world applications. By focusing on Mel-spectrogram enhancement, Mel-FullSubNet provides a novel approach that benefits both speech clarity and ASR accuracy, making it a valuable reference for further research in speech processing technology.

==== Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., & Poria, S. (2019). Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). arXiv:1906.01815v1. ====
* '''Summary:''' The paper introduces a novel approach to sarcasm detection by leveraging multimodal data. Recognizing that sarcasm often involves incongruities not just in text but also in vocal tone and facial expressions, the authors propose the first dataset, MUStARD, for sarcasm detection using audio-visual cues alongside textual data. This dataset, compiled from popular TV shows, is annotated for sarcasm, aiming to facilitate the development of models that can better understand sarcasm through the integration of multiple modes of communication.
* '''RQ:''' How can incorporating multimodal cues (textual, audio, and visual) improve the automatic classification of sarcasm compared to relying on textual data alone?
* '''Hypothesis:''' The paper hypothesizes that the inclusion of multimodal information (audio and visual cues, along with textual data) can significantly enhance the performance of sarcasm detection models, reducing the relative error rate by up to 12.9% in F-score when compared to models that use only individual modalities.
* '''Conclusion''': The research demonstrates that multimodal models significantly outperform unimodal variants in sarcasm detection, with a notable reduction in error rate. The findings underscore the importance of considering multiple communication cues, beyond just text, for effectively identifying sarcasm. The MUStARD dataset is also introduced as a valuable resource for future research in multimodal sarcasm detection.
* '''Critical Observations:'''

# Sarcasm detection benefits from multimodal analysis, including textual, audio, and visual data, highlighting the complex nature of sarcasm as a communicative act that often relies on the interplay of various signals.
# The MUStARD dataset fills a critical gap in research resources, providing a foundation for exploring how different modalities contribute to the understanding of sarcasm.
# The study's methodology, focusing on a balanced dataset and robust multimodal feature extraction techniques, sets a precedent for future work in this area.

* '''Relevance:''' This research is highly relevant to my thesis topic. It pushes the boundaries of sarcasm detection by moving beyond text analysis to include audio and visual cues, offering insights into more holistic approaches to understanding human communication. The findings and the MUStARD dataset can significantly impact the development of more nuanced and effective computational models for detecting sarcasm and other complex emotional or figurative language use cases.

==== Zhang, Yazhou, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. “CMMA: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations,” n.d. ====
* '''Summary:''' This study introduces the CMMA dataset for benchmarking multi-affection detection in Chinese multi-modal conversations, focusing on sentiment, emotion, sarcasm, and humor. The dataset comprises annotations from a variety of TV series to reflect diverse affective expressions and supports both single-task and multi-task learning paradigms for affective computing research.
* '''RQ:''' How multi-modal cues and conversational context influence the detection of multiple affects, including sentiment, emotion, sarcasm, and humor, in Chinese multi-party conversations?
* '''Hypothesis:''' Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations" likely centers on the premise that incorporating multi-modal data (text, video, audio) and conversational context significantly improves the accuracy and effectiveness of detecting multiple affects (sentiment, emotion, sarcasm, humor) in multi-party conversations. The study posits that the interplay between different modalities and the contextual understanding of conversations enhances the model's ability to interpret complex human affective expressions.
* '''Conclusion:''' The findings demonstrate that conversational context and multi-modal data significantly enhance affect detection tasks. The study also highlights the importance of multi-affect annotation for understanding complex human communications, suggesting the CMMA dataset as a valuable resource for future affective computing research.
* '''Critical observations:''' While the dataset offers comprehensive insights into multi-affect detection, its focus on Chinese TV series may limit its applicability across different linguistic and cultural contexts. Additionally, the inherent subjectivity of affect annotation poses challenges to achieving unbiased affect detection.
* '''Relevance:''' This study is pertinent to my thesis as it provides an opportunity to delve into how various feature fusion methods impact the accuracy of sarcasm recognition in Mandarin using multimodal data. Additionally, the CMMA dataset is highly beneficial to my research because it is among the few Chinese datasets that include labels for sarcasm, offering a valuable resource for studying sarcasm recognition within Mandarin-specific contexts using multimodal information.

==== Patel, T., & Scharenborg, O. (2024). Improving End-to-End Models for Children’s Speech Recognition. ''Applied Sciences'', ''14''(6), 2353. ====
* '''Summary:''' Children’s Speech Recognition (CSR) is challenging due to variable speech patterns and limited annotated data. We aim to enhance CSR when no child speech data is available. Traditionally, Vocal Tract Length Normalization (VTLN) mitigates acoustic mismatch in hybrid systems, while End-to-End (E2E) systems use data augmentation. We investigate speed perturbations, spectral augmentation, and VTLN in E2E CSR systems across Dutch, German, and Mandarin. Our experiments show that speed perturbations and spectral augmentation significantly improve performance, with VTLN offering further enhancements while maintaining adult speech recognition. VTLN benefits both genders and is particularly effective for younger children.
* '''RQ:''' How to enhance SCR performance while maintaining performance on adults’ speech when adapting the model to children’s speech?
* '''Hypothesis:''' VLTN, speed perturbation, and spectral augmentation can be useful.
* '''Conclusion:''' VLTN is used for the 1st times to improve E2E CSR work  augmentation and normalization enhance CSR task performance  the performance of adult speech is largely preserved  similar observations in all 3 languages
* '''Critical observations:''' Because VTLN needs to be trained independently and then used as a processing step after feature extraction to warp the features for training the ASR network architecture, it may not be compatible with architectures that utilize raw waveform data rather than features. As a result, integrating VTLN into such architectures requires further exploration.
* '''Relevance:''' The study's focus on improving Automatic Speech Recognition (ASR) for children's speech, despite limited annotated data, holds relevance to the endeavor of enhancing ASR performance for older adults. Both populations present challenges due to variability in speech patterns and the scarcity of annotated data. Techniques explored in the study, such as Vocal Tract Length Normalization (VTLN) and data augmentation, offer potential solutions that could be adapted to address age-related changes in older adults' speech. Comparative analyses across languages and considerations of age and gender factors provide valuable insights applicable to developing tailored ASR systems for the older adult population. Overall, the study's methodologies and findings offer valuable parallels and considerations for researchers aiming to improve ASR performance for older adults.

==== Geng, M., Xie, X., Liu, S., Yu, J., Hu, S., Liu, X., & Meng, H. (2022). Investigation of data augmentation techniques for disordered speech recognition. ''arXiv preprint arXiv:2201.05562''. ====
* '''Summary:''' The final speaker adapted system constructed using the UASpeech corpus and the best augmentation approach based on speed perturbation produced up to 2.92% absolute (9.3% relative) word error rate (WER) reduction over the baseline system without data augmentation, and gave an overall WER of 26.37% on the test set containing 16 dysarthric speakers.
* '''RQ:''' systematically investigate different data augmentation techniques for disordered speech recognition.
* '''Conclusion:''' It suggests that speed-perturbation based augmentation produces the largest improvement in system performance despite the huge mismatch between normal and disordered speech.
* '''Critical observations:'''  They increased the amount of speed perturbation data to four times and six times, with only dysarthric speech being processed, the mean WER showed that four times the amount of the original data made the model performance better than six times (4x: 29.47, 6x: 29.52). More augmented data cannot further improve the model performance. In addition, increasing the augmented data from two to four times only reduced the WER by 0.2%. They did not further increase the amount of augmented data, while according to the results when only dysarthric speech data was augmented, it is doubtful whether more data can still lower the WER. This can be explored in future studies by increasing the amount of augmented data from one to six or more times while keeping all other factors the same.
* '''Relevance:''' The study exploring data augmentation techniques for dysarthric speech recognition offers insights applicable to improving ASR performance for older adults. By addressing challenges common to both dysarthric speech and speech from older adults, such as variations in speech patterns and articulation, the study provides valuable methodologies and findings. Specifically, the effectiveness of techniques like speed perturbation-based augmentation in enhancing ASR performance underscores their potential utility in optimizing systems for recognizing older adult speech. Furthermore, the study's identification of augmentation limitations and suggestions for future research pave the way for continued refinement of ASR systems tailored to the unique characteristics of older adult speech.