Editing Multimodal Speech Recognition (section)

=== 20th Century: Early Experiments with Lip Reading ===
The idea of multimodal speech recognition can be traced back to the mid-20th century. Around this time, some pioneering researchers set out to experiment with the possibility of improving the accuracy of speech recognition in challenging acoustic environments by combining it with other modalities, which laid the foundation of multimodal approaches of ASR (Automatic [[wikipedia:Speech_recognition|Speech Recognition]]). 

As early as 1984, scholars conducted some research on [[wikipedia:Automated_Lip_Reading|automated lip reading]] to enhance speech recognition.<ref>Petajan, E. (1984). Automatic Lipreading to Enhance Speech Recognition (Speech Reading).</ref>  Prominent scholars such as Petajan, E.D. are renowned for developing one of the first audio-visual recognition systems. In his experiment, binary mouth image were used to extract mouth parameters like height, width and the area of mouth of the speaker which would be later used in the recognition system. Then the speech is processed by the acoustic recognizer first, and then passed on to the visual recognizer for final decision.<ref>Petajan, E., Bischoff, B., Bodoff, D., & Brooke, N. M. (1988, May). An improved automatic lipreading system to enhance speech recognition. In ''Proceedings of the SIGCHI conference on Human factors in computing systems'' (pp. 19-25).</ref> This visual analysis system was later used by Goldschen<ref>Goldschen, A.J., Garcia, O.N., Petajan, E.D. (1997). Continuous Automatic Speech Recognition by Lipreading. In: Shah, M., Jain, R. (eds) Motion-Based Recognition. Computational Imaging and Vision, vol 9. Springer, Dordrecht. <nowiki>https://doi.org/10.1007/978-94-015-8935-2_14</nowiki></ref> to recognize continuous speech visually. The significant contributions made by those forerunners pave the way for the integration of audio and visual information in the process of speech recognition.