Editing Multimodal Speech Recognition (section)

=== Late 20th Century - Early 21st Century: Integration of Audio and Visual Information ===
Around the 20th century, there was a growing emphasis on enhancing the [[wikipedia:Robustness|robustness]] of speech recognition systems in the face of various types of background noise in audio channel. This development gained significant attention because speech recognition systems experienced notable performance setbacks when operating in noisy environments, dealing with unfavorable acoustic channel conditions, or contending with issues like [[wikipedia:Crosstalk|crosstalk]]. Moreover, researchers observed that some amount of [[wikipedia:Orthogonality|orthogonality]] between the audio and video channels, presenting an opportunity to enhance recognition efficiency by integrating both channels. Therefore, two different approaches to combine audio and visual information have been tried.<ref>Verma, A., Faruquie, T., Neti, C., & Basu, S. (n.d.). ''LATE INTEGRATION IN AUDIO-VISUAL CONTINUOUS SPEECH RECOGNITION''.</ref>

* '''Early Integration:''' For the first approach, audio and visual features had to be computed from the acoustic and visual speech data respectively, after this, they are combined before the recognition experiment. However, this approach had a drawback, that it couldn't handle different categories or types of information in audio and video as it uses a common recognizer for both of them. <ref>Tsuhan Chen, & Rao, R. R. (1998). Audio-visual integration in multimodal communication. ''Proceedings of the IEEE'', ''86''(5), 837–852. <nowiki>https://doi.org/10.1109/5.664274</nowiki></ref>
* '''Late Integration:''' The other approach, known as late Integration, uses separate systems for audio and video recognition. Then it merges the results from these two systems to produce the final outcome. This approach is well-equipped to handle diverse categories of information in audio and video because it keeps them separate until the very end when they are combined. <ref>Bregler, C., Manke, S., Hild, H., & Waibel, A. (1993, March). Bimodal sensor integration on the example of'speechreading'. In ''IEEE International Conference on Neural Networks'' (pp. 667-671). IEEE.</ref>
When it comes to the early 21st century, some researchers introduced innovative methods, including composite [[wikipedia:Feature_(machine_learning)|feature]] [[wikipedia:Vector|vectors]] and a [[Hidden Markov Models|hidden Markov model]] structure accommodating audio-visual [[wikipedia:Asynchrony|asynchrony]].<ref>Tomlinson, M. J., Russell, M. J., & Brooke, N. M. (1996). Integrating audio and visual information to provide highly robust speech recognition. ''1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings'', ''2'', 821–824 vol. 2. <nowiki>https://doi.org/10.1109/ICASSP.1996.543247</nowiki></ref> These techniques demonstrated substantial improvements in recognition accuracy, particularly in the presence of interfering noise, as well as marked the inception of multimodal approaches, where audio and visual information converged, heralding a new era in speech recognition technology, characterized by increased accuracy and [[wikipedia:Resilience|resilience]] in diverse communication scenarios.