Editing Multimodal Speech Recognition (section)

== Key Innovations ==
Multimodal speech recognition incorporates multiple modes of input (like audio, visual, and sometimes even tactile) to interpret and convert spoken language into text or execute commands. Compared to single-modal (single mode, typically audio) speech recognition systems, multimodal systems have presented several innovations:

=== Data Fusion ===
The primary innovation in multimodal speech recognition is Data fusion.By merging information or data from multiple sensory modalities, such as audio, visual, tactile, and others, the speech recognition system can better comprehend and process speech input，and therefore improve recognition accuracy.  For example, the effective integration of both audio and visual cues, specifically speaker's lip images, can enhance ASR accuracy, particularly in challenging, noisy environments<ref>Tomlinson, M. J., Russell, M. J., & Brooke, N. M. (1996). Integrating audio and visual information to provide highly robust speech recognition. ''1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings'', ''2'', 821–824 vol. 2. <nowiki>https://doi.org/10.1109/ICASSP.1996.543247</nowiki></ref>. Fusing data from multiple sensory modalities even enables multimodal speech recognition systems to support more complex application scenarios, such as sign language recognition, gesture recognition, emotion analysis, and more, enhancing the system's versatility and adaptability. 

=== Robustness in Noisy Environments ===
One of the significant challenges in automatic speech recognition (ASR) is its performance degradation in noisy environments. Traditional ASR systems excel in quiet conditions but struggle when confronted with background noise. However, Several researches have shown that multimodal speech recognition systems perform better over single-modal speech recognition systems  in noisy environments<ref>Chibelushi, C. C. (1996). Design issues for a digital audio-visual integrated database. ''IEE Colloquium on Integrated Audio-Visual Processing for Recognition, Synthesis and Communication'', ''1996'', 7–7. <nowiki>https://doi.org/10.1049/ic:19961151</nowiki></ref><ref>Stewart, D., Seymour, R., Pass, A., & Ji Ming. (2014). Robust Audio-Visual Speech Recognition Under Noisy Audio-Video Conditions. ''IEEE Transactions on Cybernetics'', ''44''(2), 175–184. <nowiki>https://doi.org/10.1109/TCYB.2013.2250954</nowiki></ref><ref>Kashiwagi, Y., Suzuki, M., Minematsu, N., & Hirose, K. (2012). Audio-visual feature integration based on piecewise linear transformation for noise robust automatic speech recognition. ''2012 IEEE Spoken Language Technology Workshop (SLT)'', 149–152. <nowiki>https://doi.org/10.1109/SLT.2012.6424213</nowiki></ref>.This innovation provides an essential breakthrough in ensuring accurate speech recognition across various real-world scenarios, where noise interference is a common occurrence.

=== Increased Contextual Understanding ===
By analyzing both the spoken word and the speaker's facial expressions or gestures, multimodal systems can better understand the context and emotion behind the speech, leading to more accurate results. A study created a database that includes various emotions expressed by people in speech-based interactions<ref>Kessous, L., Castellano, G., & Caridakis, G. (2010). Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. ''Journal on Multimodal User Interfaces'', ''3''(1–2), 33–48. <nowiki>https://doi.org/10.1007/s12193-009-0025-5</nowiki></ref>. By combining facial expressions, body gestures, and speech analysis, they successfully integrated three modalities to enhance emotion recognition. The results show that multimodal approaches have a significant advantage in emotion recognition, especially when combining facial expressions, body gestures, and speech information. 

=== Neural Networks and Deep Learning algorithms ===
Neural Networks and Deep Learning algorithms empower multimodal speech recognition systems to extract intricate patterns and relationships from diverse data sources, including audio, video, and text. Consequently, more precise and context-aware speech recognition results can be delivered even in noisy environments, thereby enhancing their robustness in real-world usage. The LAS model (listen, attend and spell) is introduced by researchers of [[wikipedia:Google|Google]]. By combining attention mechanisms with [[wikipedia:Recurrent_neural_network|recurrent neural networks (RNNs)]], this model can significantly improve the accuracy of ASR and become a fundamental building block for multimodal speech recognition that takes both audio and visual cues into account. According to the investigation of this model, on a subset of the Google voice search task, LAS achieves a [[wikipedia:Word_error_rate|word error rate (WER)]] of 14.1% without a [[wikipedia:Dictionary|dictionary]] or a [[wikipedia:Language_model|language model]], and 10.3% with language model rescoring over the top 32 beams.<ref>Chan, W., Jaitly, N., Le, Q. V., & Vinyals, O. (2015). Listen, attend and spell. ''arXiv preprint arXiv:1508.01211''.</ref>

Moreover, with more data points from multiple input modes, multimodal systems can adapt and learn from user interactions more effectively than unimodal systems.Therefore, the field of multimodal speech recognition has driven advancements in machine learning algorithms, including deep learning models that can process and integrate multiple modalities simultaneously. CD-DNN-HMM, a state-of-the-art acoustic modeling technique, has emerged recently and demonstrated remarkable performance gains compared to the older Gaussian-mixture model-based [[wikipedia:HMM|HMMs]] (GMM-HMMs) in multiple ASR tasks <ref>Yao, K., Yu, D., Seide, F., Su, H., Deng, L., & Gong, Y. (2012). Adaptation of context-dependent deep neural networks for automatic speech recognition. ''2012 IEEE Spoken Language Technology Workshop (SLT)'', 366–369. <nowiki>https://doi.org/10.1109/SLT.2012.6424251</nowiki></ref>.