Editing Deep Learning Revolution (section)

== Key Innovations ==
Modern deep learning can be characterized by several key innovations.

=== Convolutional Neural Networks (CNNs) ===
One of the most influential network architectures in computer vision is the Convolutional Neural Network (CNN). While modern CNNs can trace their origins to Yann LeCun’s LeNet-5, initially designed for digit recognition, versions like AlexNet and ResNet have evolved to be more sophisticated, achieving significant success in image classification challenges.<ref name=":02" /> Although these CNNs are primarily associated with image processing, they also find great success in use of speech recognition systems, namely spectrograms of audio data. Spectrograms are 2D visual representations of the audio's frequency content over time, which are images that can be processed by CNNs. CNNs can extract data from these spectrograms, such as frequency patterns, phonemes, and other acoustic features.<ref>Abdel-Hamid, O., Mohamed, A. R., Jiang, H., & Penn, G. (2012, March). Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition. In ''2012 IEEE international conference on Acoustics, speech and signal processing (ICASSP)'' (pp. 4277-4280). IEEE.</ref><ref>Abdel-Hamid, O., Mohamed, A. R., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014). Convolutional neural networks for speech recognition. ''IEEE/ACM Transactions on audio, speech, and language processing'', ''22''(10), 1533-1545.</ref>

=== Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) ===
RNNs are a category of neural networks designed for sequential data.<ref>Arisoy, E., Sethy, A., Ramabhadran, B., & Chen, S. (2015, April). Bidirectional recurrent neural network language models for automatic speech recognition. In ''2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'' (pp. 5421-5425). IEEE.</ref><ref>Graves, A. (2013). Generating sequences with recurrent neural networks. ''arXiv preprint arXiv:1308.0850''.</ref> RNNs are particularly useful in ASR due to their ability model longer-distance context than word n-gram models. LSTMs, a specific type of RNN, aim to deal with the vanishing gradient issues present in traditional RNNs.<ref>Graves, A., & Graves, A. (2012). Long short-term memory. ''Supervised sequence labelling with recurrent neural networks'', 37-45.</ref> LSTMs particularly excel in the handling and modeling of sequential data, and speech signals are highly sequential. They can model the probabilistic relationships between words in a sentence, which aids in improving the recognition accuracy.

=== Transformers and Attention Mechanisms ===
Traditional RNNs and LSTMs process sequences step by step, making them slower for some applications. The transformer architecture, introduced by Vaswani et al., uses a mechanism called “attention” to weigh the importance of different parts of an input sequence when generating an output sequence. This structure allows for parallel processing of sequences, resulting in substantially reduced training times compared to earlier RNNs and LSTMs.<ref>Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. ''Advances in neural information processing systems'', ''30''.</ref> The transformers and attention mechanisms are especially useful in handling long audio sequences and complex language modeling.

=== Generative Adversarial Networks (GANs) ===
GANs consist of two networks: a generator that creates data and a discriminator that evaluates it. GANs are not usually directly used in speech recognition, however, they are used successfully in generating realistic audio data. GANs have the capability to produce remarkably realistic synthetic data, ranging from art pieces to high-resolution images.<ref>Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. ''Advances in neural information processing systems'', ''27''.</ref> GANs have proven particularly useful in the training and modeling of accent adaptation of ASR systems. GANs are also useful in audio enhancement and denoising.

=== Transfer Learning and Pre-trained Models ===
Deep learning models, especially those used for NLP tasks, can have millions or even billions of parameters. Training such models from scratch requires extensive computational resources. Transfer learning circumvents this by leveraging pre-trained models. These models, trained on vast datasets, can be fine-tuned with a smaller amount of task-specific data, accelerating development and boosting performance.<ref>Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. ''arXiv preprint arXiv:1810.04805''.</ref> <ref>Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.</ref> Transfer learning and pre-trained models have been crucial to the enhancement of ASR systems, primarily by offering significant improvements in accuracy, efficiency, and the ability to adapt models to various tasks.