Vocoder Development: Difference between revisions
| Line 12: | Line 12: | ||
==== '''Mechanical and Electro-Mechanical Era''' ==== | ==== '''Mechanical and Electro-Mechanical Era''' ==== | ||
This era commenced with early speech synthesis attempts in the late 18th century. In 1769, [[Wolfgang von Kempelen's Speaking Machine (1769)|Wolfgang von Kempelen]] engineered a mechanical speaking device that ingeniously emulated speech sounds through the mechanical simulation of vocal cords, vocal tracts, and lungs. Concurrently, [[Mechanical synthesis|Christian Kratzenstein]] embarked on early mechanical synthesis by employing five organ pipe-like resonators to successfully reproduce five distinct vowels. Later in 1824, Wheatstone introduced a "Speaking Machine," and in 1846, Joseph Faber constructed "Euphonia," both utilizing a series of mechanical components to produce speech-like sounds. During this period, the mechanical and electro-mechanical era of speech synthesis was drawing to a close. Despite their constraints in generating an extensive vocabulary and sentences, these early endeavors made noteworthy contributions to the exploration of speech synthesis, igniting extensive research into the physiology of speech production and experimental phonetics. The comprehension of acoustic resonators, spectral components, and formants marked a pivotal shift toward the scientific investigation of human sound production, setting the stage for the subsequent era of speech synthesis <ref>Schroeder, M. R. (1993). [https://doi.org/10.1016/0167-6393(93)90074-U A brief history of synthetic speech]. ''Speech Communication'', ''13''(1–2), 231–237. </ref><ref name=":1">Story, B. H. (2019). [https://www.researchgate.net/profile/Brad-Story/publication/333614830_History_of_Speech_Synthesis_Chapter_1_Routledge_Handbook_of_Phonetics_W_Katz_and_P_Assmann_Eds_2019/links/5e65a26392851c7ce05353c6/History-of-Speech-Synthesis-Chapter-1-Routledge-Handbook-of-Phonetics-W-Katz-and-P-Assmann-Eds-2019.pdf History of speech synthesis]. ''Teoksessa The Routledge Handbook of Phonetics, toimittaneet William F. Katz & Peter F. Assmann'', 9-33. </ref>. | This era commenced with early speech synthesis attempts in the late 18th century. In 1769, [[Wolfgang von Kempelen's Speaking Machine (1769)|Wolfgang von Kempelen]] engineered a mechanical speaking device that ingeniously emulated speech sounds through the mechanical simulation of vocal cords, vocal tracts, and lungs. Concurrently, [[Mechanical synthesis|Christian Kratzenstein]] embarked on early mechanical synthesis by employing five organ pipe-like resonators to successfully reproduce five distinct vowels. Later in 1824, [[Wheatstone's "Speaking Machine" (1824)|Wheatstone]] introduced a "Speaking Machine," and in 1846, Joseph Faber constructed "Euphonia," both utilizing a series of mechanical components to produce speech-like sounds. During this period, the mechanical and electro-mechanical era of speech synthesis was drawing to a close. Despite their constraints in generating an extensive vocabulary and sentences, these early endeavors made noteworthy contributions to the exploration of speech synthesis, igniting extensive research into the physiology of speech production and experimental phonetics. The comprehension of acoustic resonators, spectral components, and formants marked a pivotal shift toward the scientific investigation of human sound production, setting the stage for the subsequent era of speech synthesis <ref>Schroeder, M. R. (1993). [https://doi.org/10.1016/0167-6393(93)90074-U A brief history of synthetic speech]. ''Speech Communication'', ''13''(1–2), 231–237. </ref><ref name=":1">Story, B. H. (2019). [https://www.researchgate.net/profile/Brad-Story/publication/333614830_History_of_Speech_Synthesis_Chapter_1_Routledge_Handbook_of_Phonetics_W_Katz_and_P_Assmann_Eds_2019/links/5e65a26392851c7ce05353c6/History-of-Speech-Synthesis-Chapter-1-Routledge-Handbook-of-Phonetics-W-Katz-and-P-Assmann-Eds-2019.pdf History of speech synthesis]. ''Teoksessa The Routledge Handbook of Phonetics, toimittaneet William F. Katz & Peter F. Assmann'', 9-33. </ref>. | ||
==== '''Electrical and Electronic Era''' ==== | ==== '''Electrical and Electronic Era''' ==== | ||
Revision as of 16:52, 15 October 2023
Introduction
A vocoder is a crucial component in the realm of speech synthesis, employed in various applications ranging from telecommunications and entertainment to assistive technologies and virtual assistants. Its ability to analyze and synthesize the complex spectrum of human speech, including the pitch, formants, and prosody, has transformed the way we interact with and harness the power of synthesized voice.
First emerged in the 1930s, the term "vocoder" is derived from "voice" and "coder", and was initially conceived for telecommunication purposes. In the 1950s, Lincoln Laboratory conducted research on detecting pitch in speech, which subsequently influenced the development of voice coders, commonly known as vocoders. These devices are designed to decrease the amount of bandwidth required for transmitting speech. This reduction in bandwidth offers two advantages: it reduces the cost of transmitting and receiving speech, while also enhancing the potential for maintaining privacy.[1]
Initially, a vocoder described a device designed to compress speech for efficient transmission over telephone lines. The idea involved splitting speech into parts using filters and then reconstructing it on the other end. The goal was to save bandwidth, but in practice, early vocoders faced challenges in preserving the natural quality of speech. Additionally, the original vocoder only transmitted loudness, not sound quality[2].
As technology advanced, the phase vocoder emerged, preserving both the loudness and sound quality of speech. The phase vocoder is a significant advancement in vocoder technology and plays a pivotal role in modern speech synthesis and audio signal processing. It was developed to address critical limitations in early vocoders and revolutionized the way we process and manipulate audio signals. [2]
Historical Context
Speech synthesis, commonly known as Text-to-Speech (TTS), has gained increasing importance in people's lives. The development of TTS technology spans centuries, beginning with the Mechanical and Electro-Mechanical Era that relied on mechanical and electro-mechanical components to simulate speech sounds. It progressed through the Electrical and Electronic Era, where electronic technology was used to create early formant synthesizers and speech coding systems, and eventually entered the digital and computational era, marked by the transition to digital signal processing and computational methods for speech synthesis.
Mechanical and Electro-Mechanical Era
This era commenced with early speech synthesis attempts in the late 18th century. In 1769, Wolfgang von Kempelen engineered a mechanical speaking device that ingeniously emulated speech sounds through the mechanical simulation of vocal cords, vocal tracts, and lungs. Concurrently, Christian Kratzenstein embarked on early mechanical synthesis by employing five organ pipe-like resonators to successfully reproduce five distinct vowels. Later in 1824, Wheatstone introduced a "Speaking Machine," and in 1846, Joseph Faber constructed "Euphonia," both utilizing a series of mechanical components to produce speech-like sounds. During this period, the mechanical and electro-mechanical era of speech synthesis was drawing to a close. Despite their constraints in generating an extensive vocabulary and sentences, these early endeavors made noteworthy contributions to the exploration of speech synthesis, igniting extensive research into the physiology of speech production and experimental phonetics. The comprehension of acoustic resonators, spectral components, and formants marked a pivotal shift toward the scientific investigation of human sound production, setting the stage for the subsequent era of speech synthesis [3][4].
Electrical and Electronic Era
In the electrical and electronic era, Text-to-Speech (TTS) technology underwent a transformative evolution, with the convergence of electricity and emerging electronic components giving rise to more efficient speech synthesis systems. This era dated back to 1922 when John Q. Stewart designed a system utilizing early electronic technology to generate speech sounds, essentially introducing the first electrical formant synthesizer. Electronic speech coding was inaugurated by Homer Dudley in response to bandwidth limitations on telegraph cables and the substantial bandwidth requirements for transmitting spectral content. Dudley designed Voder in 1930 and later created the first analysis-synthesis system, the Vocoder, in 1940s, introducing the concept of analyzing speech into spectral components and synthesizing it, with the process of extracting low-frequency spectral data from an acoustic speech signal and transmitting it via a low-bandwidth cable, and modulating a locally generated carrier signal at the receiving end[4]. This laid the foundation for subsequent TTS developments and paved the way for further research in the field of speech synthesis. However, the Vocoder had limitations in terms of naturalness and intelligibility in speech synthesis, primarily due to its complexity and dependence on manual tuning, making it challenging to achieve high-quality and versatile speech synthesis [4].
Key Innovations
Innovations in vocoders are often developments in deep learning, signal processing and machine learning.
According to Dudley, the use of vocoders are advantageous for more secure communications and a greater number of telephone channels in the same frequency space. Digital communication made eavesdropping conversations more difficult. However, the digitization of speech required wider transmission bandwidths. The vocoder was used to reduce the speech bitrate to a number that could be handled through the average telephone channel. [1]
Pitch detection in vocoding was a difficulty in the late 1950s. This is necessary to synthesize speech of acceptable quality. [5] Speech is created by vibrating the vocal folds, the air that is moving and closing the mouth. The vocal fold vibrations are analyzed to find the pitch. To estimate the fundamental frequency, the Lincoln Laboratory designed an algorithm that used parallel processing, which produced accurate results. [1]
In a channel vocoder, formant tracking helps recreate speech. It looks at the first four formant features to estimate the speech sounds. This method can make a vocoder use less data or bandwidth and still sound good. [6]
Another innovation is the speaker-dependent WaveNet vocoder that used existing vocoder data as additional information. This method does not need to explicitly model certain aspects of speech and offers better sound quality. It successfully recovers lost information and captures source details more accurately compared to traditional vocoders. [7]
WORLD, a vocoder-based speech synthesis system, was developed in an effort to improve the sound quality of real time applications using speech. Real time processing has been difficult because of the high computational costs. This new system has a good sound quality and is also quick in processing. [8]
Impact
Wenjun
Future Research
Despite the significant advancements in technology and the development of new speech synthesis systems, ranging from DECtalk in the 1980s to the cutting-edge AI models of today, vocoders continue to play a key role in various applications. Today, vocoders are integral components of state-of-the-art speech synthesis systems, including WORLD[9], designed specifically for real-time applications, and BigVGAN[10] , which uses the power of generative adversarial networks (GANs[11]). A majority of contemporary vocoders rely on neural networks for their operation, but improvements could be made in this direction too. Additionally, the integration of generative AI holds the promise of further enhancing the quality of vocoder-synthesized voices.
Furthermore, vocoders are largely used in the field of music production, although we have yet to achieve the creation of truly authentic singing voices through them. The potential to explore and refine the use of vocoders in this context offers a pathway to a broader advancement in speech synthesis technology.
Additionally, vocoders are currently deployed to address other challenges in voice technology. Vocoder-synthesized voices serve as tools in training noise-robust models[12] and detect fake audio[13].
LLM Review
Erin
References
- ↑ 1.0 1.1 1.2 Gold, Bernard. “Gold A History of Vocoder Research at Lincoln Laboratory.”, The Lincoln Laboratory Journal. Volume 3. Number 2 (1990)
- ↑ 2.0 2.1 Gordon, John William, and John Strawn. An introduction to the phase vocoder No. 55. CCRMA, Department of Music, Stanford University, 1987.
- ↑ Schroeder, M. R. (1993). A brief history of synthetic speech. Speech Communication, 13(1–2), 231–237.
- ↑ 4.0 4.1 4.2 Story, B. H. (2019). History of speech synthesis. Teoksessa The Routledge Handbook of Phonetics, toimittaneet William F. Katz & Peter F. Assmann, 9-33.
- ↑ Gold, B., & Rabiner, L. (1969). Parallel processing techniques for estimating pitch periods of speech in the time domain. The Journal of the Acoustical Society of America, 46(2B), 442-448.
- ↑ Gold, B. (1980, April). Formant representation of parameters for a channel vocoder. In ICASSP'80. IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 5, pp. 128-130). IEEE.
- ↑ Tamamori, A., Hayashi, T., Kobayashi, K., Takeda, K., & Toda, T. (2017, August). Speaker-dependent wavenet vocoder. In Interspeech (Vol. 2017, pp. 1118-1122).
- ↑ Morise, M., Yokomori, F., & Ozawa, K. (2016). WORLD: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 99(7), 1877-1884.
- ↑ Morise, Masanori, Fumiya Yokomori, e Kenji Ozawa. «WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications». IEICE Transactions on Information and Systems E99.D, fasc. 7 (2016): 1877–84. https://doi.org/10.1587/transinf.2015EDP7457.
- ↑ Lee, Sang-gil, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. «BigVGAN: A Universal Neural Vocoder with Large-Scale Training». arXiv, 16 february 2023. http://arxiv.org/abs/2206.04658.
- ↑ Rocca, Joseph. "Understanding generative adversarial networks (gans)." Medium 7 (2019): 20.
- ↑ Zheng, Nengheng, Yupeng Shi, Yuyong Kang, e Qinglin Meng. «A Noise-Robust Signal Processing Strategy for Cochlear Implants Using Neural Networks». In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8343–47, 2021. https://doi.org/10.1109/ICASSP39728.2021.9413452.
- ↑ Yan, Xinrui, Jiangyan Yi, Jianhua Tao, Chenglong Wang, Haoxin Ma, Tao Wang, Shiming Wang, e Ruibo Fu. «An Initial Investigation for Detecting Vocoder Fingerprints of Fake Audio». In Proceedings of the 1st International Workshop on Deepfake Detection for Audio Multimedia, 61–68. DDAM ’22. New York, NY, USA: Association for Computing Machinery, 2022. https://doi.org/10.1145/3552466.3556525.
Team Members
Alice Vanni, Amber, Chenyi, Erin Shi, Wenjun