Large Vocabulary Continuous Speech Recognition: Difference between revisions

From MSc Voice Technology
Jump to navigation Jump to search
Line 19: Line 19:
=== Multimodal integration in speech recognition ===
=== Multimodal integration in speech recognition ===


Multimodal integration in speech recognition refers to the process of combining information from multiple sensory modalities, such as audio (speech signals) and visual (lip movements or facial expressions), to improve the accuracy and robustness of speech recognition systems.
[[Multimodal integration in speech recognition]] refers to the process of combining information from multiple sensory modalities, such as audio (speech signals) and visual (lip movements or facial expressions), to improve the accuracy and robustness of speech recognition systems.


In traditional speech recognition, the system relies solely on audio input to transcribe spoken words.  However, this approach can be limited in noisy environments or when there are variations in speech articulation.  Multimodal integration seeks to address these limitations by incorporating additional sources of information, such as visual cues from the speaker's mouth movements.
In traditional speech recognition, the system relies solely on audio input to transcribe spoken words.  However, this approach can be limited in noisy environments or when there are variations in speech articulation.  Multimodal integration seeks to address these limitations by incorporating additional sources of information, such as visual cues from the speaker's mouth movements.

Revision as of 21:11, 17 September 2023

Introduction

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Historical Context

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Key Innovations

Development of End-to-End Models

All the time, LVCSR is developing in a more end-to-end direction. The traditional LVCSR usually includes various modules, for instance, acoustic models, language models, and pronunciation dictionaries. However, recently decade, to simplify the process and improve LVCSR efficiency, experts are focusing on how to use a neural network to couple all these modules into a unite model.

This approach utilizes a Recurrent Neural Network (RNN) model for phoneme recognition, which is the process of converting spoken sounds into text. A distinctive feature of this novel method is its ability to perform this conversion without the need for complex alignment operations, allowing for the direct generation of the desired textual output. Additionally, it is capable of operating in real-time scenarios where there are stringent timing requirements.[1]

The foundation of this method is an extension of a neural machine translation model, which bears similarities to certain existing speech recognition techniques. However, what sets it apart is its capacity to calculate scores for all positions within both the input and output sequences, subsequently employing these scores to aid in the recognition process. An innovative aspect of this approach is the explicit utilization of these scores for alignment purposes, facilitating the generation of accurate textual representations. Furthermore, the decoder state in this model incorporates information regarding prior alignment choices, enhancing the precision of speech understanding.

A key advantage of this method is its ability to perform decoding in an almost deterministic manner, making it suitable for real-time speech recognition without the need for computationally intensive procedures. This performance characteristic suggests the feasibility of extending the method to large-vocabulary speech recognition systems. It also opens up the possibility of directly searching for the most probable word sequences, as opposed to conducting searches at the phoneme or frame level, as is traditionally done in Hidden Markov Model (HMM)-based hybrid systems. This feature contributes significantly to improving the speed and accuracy of speech recognition.

Multimodal integration in speech recognition

Multimodal integration in speech recognition refers to the process of combining information from multiple sensory modalities, such as audio (speech signals) and visual (lip movements or facial expressions), to improve the accuracy and robustness of speech recognition systems.

In traditional speech recognition, the system relies solely on audio input to transcribe spoken words.  However, this approach can be limited in noisy environments or when there are variations in speech articulation.  Multimodal integration seeks to address these limitations by incorporating additional sources of information, such as visual cues from the speaker's mouth movements.

The idea behind multimodal integration is that different modalities can provide complementary information that helps disambiguate spoken words.  For example, lip movements can provide information about the shape of the mouth and the position of the tongue, which can be useful for disambiguating similar-sounding words.[3]


Impact

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Future research

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. [2]

References

To insert a reference, type <ref> and paste the source your exported from Zotero (or whatever reference manager you're using) in the pop up box which appears. Make sure links in citations are clickable using proper formatting. Once you do this, a footnote will appear.[3] and a reference comes at the end automatically. Please use this method to cite for Wiki articles only, not for your thesis.

  1. [1]
  2. Feng, S., Kudina, O., Halpern, B. M., & Scharenborg, O. (2021). Quantifying Bias in Automatic Speech Recognition (arXiv:2103.15122). arXiv. http://arxiv.org/abs/2103.15122
  3. Glantz, Richard "SHOEBOX: a personal file handling system for textual data." In Proceedings of the November 17-19, 1970, Fall Joint Computer Conference 1970. 535-545. [2]


group members: Yan Liao, Jingxuan Yue, Chenyu Li