Editing Multimodal Speech Recognition (section)

=== Multimodal Speech Recognition's Influences on Speech Technology ===
Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), which are two key [[Introduction of Voice Assistants|deep learning]] methods. In multimodal speech recognition, audio and image data play a vital role in extracting and merging features. A common strategy is to use CNNs to process both audio and image data, extracting important features. For audio, CNNs can create spectrograms, while for images, they identify objects and generate feature vectors capturing important visual details. After feature extraction, models like RNNs or transformers are used to integrate these features, capturing the connections between audio and visual data. Training audio recognition and visual recognition separately linking the two together resulted in lower phone error rates. however, when a new bilinear DNN network was used, which allowed training both audio and visual, the result was to achieve even lower error rates.<ref>Mroueh, Y., Marcheret, E., & Goel, V. (2015). Deep multimodal learning for Audio-Visual Speech Recognition. ''2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'', 2130–2134. <nowiki>https://doi.org/10.1109/ICASSP.2015.7178347</nowiki></ref>

CTC(Connectionist Temporal Classification) is a technique for aligning audio and text, and it has found application in multimodal speech recognition. CTC allows for the alignment of audio and text data, enabling the system to understand when and how to correspond audio features with text labels. An experiment tested the CTC bidirectional LSTM acoustic model and the S2S visual semantic feature model. As a result, the CTC output is closer to the acoustics of an utterance.<ref>Palaskar, S., Sanabria, R., & Metze, F. (2018). End-to-end Multimodal Speech Recognition. ''2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'', 5774–5778. <nowiki>https://doi.org/10.1109/ICASSP.2018.8462439</nowiki></ref>

Multimodal speech recognition systems need to adapt to varying environmental noise conditions, allowing the system to automatically adjust in different environments, thereby enhancing robustness. Also, for speaker adaptation, models also need to adapt to the speech characteristics of different speakers. Adaptive techniques can capture the speaker-specific speech traits, improving recognition performance. The experiment has shown that the MMASR model shows significant gains (up to 4.2% WER improvements) compared to traditional speech-to-text architecture in noisy environments.<ref>Srinivasan, T., Sanabria, R., & Metze, F. (2019). ''Analyzing Utility of Visual Context in Multimodal Speech Recognition Under Noisy Conditions''. <nowiki>https://doi.org/10.48550/ARXIV.1907.00477</nowiki></ref>

Traditional Multimodal Speech Recognition systems comprise many independent processing stages, including audio extraction, image extraction, and speech recognition. Each stage has individual design, which may lead to more system complexity, high computational costs. The key advantage of [[Development of End-to-End Models|end-to-end]] multimodal models lies in simplifying the entire multimodal speech recognition pipeline while enhancing performance. End-to-end models take audio and image data as inputs into a single deep learning model, enabling the model to automatically learn how to extract speech information from multimodal inputs. Understanding speech from visual signals alone has been of interest for decades. One researcher proposed a multimodal attention method to get information from multimodal input. Modality attention mechanism is integrated in an end-to-end attention based AVSR system. The results show that our proposed method obtains a 36% improvement.<ref>Zhou, P., Yang, W., Chen, W., Wang, Y., & Jia, J. (2019). Modality Attention for End-to-end Audio-visual Speech Recognition. ''ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'', 6565–6569. <nowiki>https://doi.org/10.1109/ICASSP.2019.8683733</nowiki></ref>