Editing State-of-the-art (section)

==== Gandhi, S., von Platen, P., & Rush, A. M. (2023). Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling. ''arXiv preprint arXiv:2311.00430''. ====

* Summary: The study introduces a novel approach to compressing pre-trained large speech recognition models for efficient deployment in low-resource environments. By leveraging large-scale pseudo-labeling, the research achieves a smaller variant, Distil-Whisper, which significantly reduces the model size and inference time without considerably sacrificing performance. This method particularly benefits low-resource languages by maintaining robustness across various acoustic scenarios and demonstrating potential in extending sophisticated ASR capabilities to languages with limited training data.
* RQ: How can the size of pre-trained speech recognition models, specifically the Whisper model, be reduced for efficient deployment in low-latency or resource-constrained environments while maintaining model robustness and performance?
* Hypothesis: By using pseudo-labelling to create a large-scale open-source dataset and applying a simple word error rate (WER) heuristic to select only the highest quality pseudo-labels for training, it is possible to distill the Whisper model into a smaller variant (Distil-Whisper) that is significantly faster and more parameter-efficient without substantially sacrificing performance.
* Conclusion: Distil-Whisper successfully demonstrates the feasibility of distilling a large-scale speech recognition model into a significantly smaller and faster version without substantial loss in performance. The distilled model achieves a WER performance within 1% of the original Whisper model on out-of-distribution test data, maintains robustness against difficult acoustic conditions, and reduces the propensity for hallucination errors in long-form audio. Furthermore, Distil-Whisper, when paired with Whisper for speculative decoding, offers a significant speed-up in inference times while ensuring identical outputs to the original model.
* Critical Observations: The approach underscores the effectiveness of large-scale pseudo-labelling and a straightforward WER-based heuristic in filtering training data for distillation purposes. The research highlights a crucial balance between model size, speed, and performance robustness, contributing to practical speech recognition applications, especially in constrained environments.
* Relevance: The methodology demonstrates potential for extending sophisticated ASR capabilities to languages with fewer resources by leveraging transfer learning and pseudo-labeling techniques.