Deep Learning Revolution: Difference between revisions
No edit summary |
|||
Line 74: | Line 74: | ||
<references /> | <references /> | ||
== | == Contributors == | ||
Yuxing (Patrick) Ouyang, Xiaoling (River) Lin, Brandi Hongell | Yuxing (Patrick) Ouyang, Xiaoling (River) Lin, Brandi Hongell |
Revision as of 14:03, 18 September 2023
Introduction
Machine learning is a science of finding patterns in data, while deep learning, a subset of machine learning inspired by human brain structures, uses layered and structured neural networks to process raw data and learn features automatically. Recent key innovations such as RNN, LSTM and Transformers have shifted the landscape of speech technology which was previously dominated by Hidden Markov Models (HMMs).
Historical Context
Deep learning is based on the concept of neural network, which was first inspired by the neural structures of the human brains. In 1962, Frank Rosenblatt published "Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms." In this work, he introduced the multilayer perceptron (MLP), often regarded as the precursor to modern deep learning.[1] In 1970, Seppo Linnainmaa introduced what we now recognize as the backpropagation algorithm, a foundational technique that underpins modern deep learning frameworks such as PyTorch and TensorFlow.[2] In 1986, David E. Rumelhart et al. published an experimental analysis of applying backpropagation to MLPs,[3] which made backpropagation more widely accepted and raised researchers’ interest in MLP. At this time, it looked like tough problems such as speech recognition could be solved.
However, early neural networks faced significant computational limitations at that age, hindering their scalability until the past decade or so. The modern rapid ascent of deep learning over the past decade or so can be mainly attributed to two factors.
The first is the exponential surge in computational power, particularly from GPUs.[4] In 2012, a defining moment was released when a deep convolutional neural network named AlexNet dramatically outperformed other methods in the ImageNet competition.[5] The primary insight from the AlexNet paper was the critical importance of a neural network's depth for achieving superior performance. This depth, however, came with computational intensity, underscoring the significance of using GPUs during training.
The second is the availability of vast data sets and related advanced big data infrastructure. In the domains of language, speech and vision, a noticeable transition emerged around 2014-2015. During this time, datasets appeared that were orders of magnitude larger than those prevalent in the preceding decade.[6]
Key Innovations
Modern deep learning can be characterized by several key innovations.
Convolutional Neural Networks (CNNs)
One of the most influential network architectures in computer vision is the Convolutional Neural Network (CNN). While modern CNNs can trace their origins to Yann LeCun’s LeNet-5, initially designed for digit recognition, versions like AlexNet and ResNet have evolved to be more sophisticated, achieving significant success in image classification challenges.[5]
Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)
RNNs are a category of neural networks designed for sequential data. RNNs are particularly useful in ASR due to their ability model longer-distance context than word n-gram models.[7] LSTMs, a specific type of RNN, aim to deal with the vanishing gradient issues present in traditional RNNs.[8] LSTMs particularly excel in the handling of sequential data, and speech signals are highly sequential.
Transformers and Attention Mechanisms
Traditional RNNs and LSTMs process sequences step by step, making them slower for some applications. The transformer architecture, introduced by Vaswani et al., uses a mechanism called “attention” to weigh the importance of different parts of an input sequence when generating an output sequence. This structure allows for parallel processing of sequences, resulting in substantially reduced training times compared to earlier RNNs and LSTMs.[9]
Generative Adversarial Networks (GANs)
GANs consist of two networks: a generator that creates data and a discriminator that evaluates it. Through their competition, GANs have the capability to produce remarkably realistic synthetic data, ranging from art pieces to high-resolution images.[10] GANs have proven particularly useful in the training and modeling of accent adaptation of ASR systems.
Transfer Learning and Pre-trained Models
Deep learning models, especially those used for NLP tasks, can have millions or even billions of parameters. Training such models from scratch requires extensive computational resources. Transfer learning circumvents this by leveraging pre-trained models. These models, trained on vast datasets, can be fine-tuned with a smaller amount of task-specific data, accelerating development and boosting performance.[11] [12]
Impact on the field
Voice technology, encompassing Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) synthesis, has been significantly impacted by some of the previous key innovations in deep learning.
Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)
RNNs and LSTMs are particularly suited for sequential data like voice, making them ideal for ASR. ASR involves converting spoken language into written text, which requires understanding temporal dependencies in spoken sequences. LSTM's ability to capture long-term dependencies in sequences has made them an essential component of many state-of-the-art ASR systems.[13]
Transformers and Attention Mechanisms
For TTS, neural networks like Tacotron 2 utilize attention mechanisms to map characters to mel-scale spectrograms.[14] Additionally, the transformer architecture's parallel processing capability has been leveraged in both ASR and TTS systems to enhance efficiency and accuracy.[15]
Generative Adversarial Networks (GANs)
In the field of TTS, GANs have been explored for generating high quality speech waveforms. Neural networks like Parallel WaveGAN use GANs to convert mel-spectrograms to raw audio waveforms, producing more natural-sounding speech.[16]
Transfer Learning and Pre-trained Models
Transfer learning, especially with large pre-trained models, is increasingly used in voice technologies. While it's more dominant in the NLP field, the principle of using knowledge from one domain and applying it to another is seeing its application in ASR and TTS. For instance, pre-trained models on vast text datasets can provide a foundational understanding of language, which can then be fine-tuned on specific voice datasets for improved performance.[17]
Multilingual Automatic Speech Recognition
Advancements in deep learning have led to innovations in the field of multilingual Automatic Speech Recognition (ASR). These advancements have facilitated the development of multilingual deep neural networks characterized by shared hidden layers across multiple languages. The output layers of the DNN’s can model either a universal phone set using the International Phonetic Alphabet or multiple sets of language specific senones.[18] These outputs have been used in many Language Adaptive Training techniques. Speech recognition systems leveraging these multilingual deep learning neural networks consistently yield substantial performance enhancements, particularly in lower-resourced languages.[19]
Low-Latency Automatic Speech Recognition
Usage of deep learning such as Amortized Neural Networks (AmNets) has allowed for advances in low-latency ASR. In one study, researchers use these AmNets to reduce processing and latency required for ASR by applying them to the Recurrent Neural Network Transducer (RNN-T).[20] These advances have allowed for ASR processing in virtually real-time without sacrificing accuracy.
Future research
Multimodal Fusion
Multimodal fusion refers to obtaining information from multiple fields, including voice, text, image and video, to improve the performance of models. One of the applications is conference summary.[21] Many employees are bothered by frequent meetings and lengthy content. And to extract conference summary, meeting text information is not sufficient. That’s why multimodal fusion takes multimodal information, such as video and audio, to make more comprehensive insight. For example, it can identify speech intonation and discern whether discussions involve emotions or disagreements.
Zero-shot and Few-shot Learning
The continued development of Deep Neural Networks allows for the potential enhancements of zero-shot and few-shot learning for ASR systems, especially in the English language. This will allow ASR’s to recognize speech with little to no training. One example of this is a study conducted on AphasiaBank, the largest datasource for aphasic speech recognition. Although it is the largest datasource, AphasiaBank only holds under 100 hours of audio data. Despite this, pre-training large models on a universal dataset shows a zero-shot 22% improvement on AphasiaBank.[22] This bodes well for other ASR applications with small resource pools.
Improved Accuracy
Accuracy of automatic speech recognition systems will continue to improve alongside the increase of training and development of deep neural networks. This is attributed to more robust model architectures as well as growing datasets of higher quality used to train these models.[23]
Personalization
Automatic speech recognition continues to become more tailored to the user experience rather than being an “out of the box” product. Currently, personalization is often done in a server-based training environment, however, this poses many issues, such as: data privacy, update delays, and computing costs.[24] Future research points towards on-device ASR personalization using limited data sets from the speaker as a possible remedy.
Privacy and Security
Along with the mass deployment of systems using ASR, namely voice assistants such as Alexa and Siri, have come security concerns with always-on microphones[25] and manipulated inputs.[26] Ongoing research is needed in order to protect this ever-evolving space.
Ethical Considerations
There are many ethical considerations when it comes to the continued development of deep learning in ASR. One prominent issue that continues to be researched is bias. Even well-trained ASR systems often face problems in large variations in speech that are due to characteristics such as: age, gender, race, speech impairments, and accents.[27] Further training and research is needed with more diverse datasets to combat this issue and make ASR’s accessible to the greater audience. Similarly, fairness will need to be taken into consideration. Fairness in ASR relates to how equally a system performs between different subgroups of a population.[28]
LLM Review
References
- ↑ Tappert, C. C. (2019, December). Who is the father of deep learning?. In 2019 International Conference on Computational Science and Computational Intelligence (CSCI) (pp. 343-348). IEEE.
- ↑ Schmidhuber, J. (2022). Annotated history of modern AI and Deep learning. arXiv preprint arXiv:2212.11279.
- ↑ Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. nature, 323(6088), 533-536.
- ↑ Sejnowski, T. J. (2018). The deep learning revolution. MIT press.
- ↑ 5.0 5.1 Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.
- ↑ Pablo Villalobos and Anson Ho (2022), "Trends in Training Dataset Sizes". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/trends-in-training-dataset-sizes' [online resource]
- ↑ Arisoy, E., Sethy, A., Ramabhadran, B., & Chen, S. (2015, April). Bidirectional recurrent neural network language models for automatic speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5421-5425). IEEE.
- ↑ Graves, A., & Graves, A. (2012). Long short-term memory. Supervised sequence labelling with recurrent neural networks, 37-45.
- ↑ Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
- ↑ Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27.
- ↑ Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- ↑ Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.
- ↑ Graves, A., Mohamed, A. R., & Hinton, G. (2013, May). Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing (pp. 6645-6649). Ieee.
- ↑ Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., ... & Wu, Y. (2018, April). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 4779-4783). IEEE.
- ↑ Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., ... & Zhang, W. (2019, December). A comparative study on transformer vs rnn in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 449-456). IEEE.
- ↑ Yamamoto, R., Song, E., & Kim, J. M. (2020, May). Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6199-6203). IEEE.
- ↑ Li, X., Wang, C., Tang, Y., Tran, C., Tang, Y., Pino, J., ... & Auli, M. (2020). Multilingual speech translation with efficient finetuning of pretrained models. arXiv preprint arXiv:2010.12829.
- ↑ Tong, S., Garner, P. N., & Bourlard, H. (2017). An investigation of deep neural networks for multilingual speech recognition training and adaptation (No. CONF, pp. 714-718).
- ↑ Huang, J. T., Li, J., Yu, D., Deng, L., & Gong, Y. (2013, May). Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In 2013 IEEE international conference on acoustics, speech and signal processing (pp. 7304-7308). IEEE.
- ↑ Macoskey, J., Strimel, G. P., Su, J., & Rastrow, A. (2021). Amortized neural networks for low-latency speech recognition. arXiv preprint arXiv:2108.01553.
- ↑ Li, M., Zhang, L., Ji, H., & Radke, R. J. (2019, July). Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 2190-2196).
- ↑ Xiao, A., Zheng, W., Keren, G., Le, D., Zhang, F., Fuegen, C., ... & Mohamed, A. (2021). Scaling asr improves zero and few shot learning. arXiv preprint arXiv:2111.05948.
- ↑ Tao, J., Evanini, K., & Wang, X. (2014, December). The influence of automatic speech recognition accuracy on the performance of an automated speech assessment system. In 2014 IEEE Spoken Language Technology Workshop (SLT)(pp. 294-299). IEEE.
- ↑ Tomanek, K., Beaufays, F., Cattiau, J., Chandorkar, A., & Sim, K. C. (2021). On-device personalization of automatic speech recognition models for disordered speech. arXiv preprint arXiv:2106.10259.
- ↑ Sun, K., Chen, C., & Zhang, X. (2020, November). " Alexa, stop spying on me!" speech privacy protection against voice assistants. In Proceedings of the 18th conference on embedded networked sensor systems (pp. 298-311).
- ↑ Abdullah, H., Warren, K., Bindschaedler, V., Papernot, N., & Traynor, P. (2021, May). Sok: The faults in our asrs: An overview of attacks against automatic speech recognition and speaker identification systems. In 2021 IEEE symposium on security and privacy (SP) (pp. 730-747). IEEE.
- ↑ Feng, S., Kudina, O., Halpern, B. M., & Scharenborg, O. (2021). Quantifying bias in automatic speech recognition. arXiv preprint arXiv:2103.15122.
- ↑ Veliche, I. E., & Fung, P. (2023, June). Improving Fairness and Robustness in End-to-End Speech Recognition Through Unsupervised Clustering. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
Contributors
Yuxing (Patrick) Ouyang, Xiaoling (River) Lin, Brandi Hongell