Advancements in AI TTS (2020s)

Introduction

Since the turn of the 21st century, AI Text-to-Speech (TTS) technology has experienced a remarkable evolution, transitioning from rule-driven synthesis to end-to-end approaches founded on deep learning. These transformative shifts have resulted in remarkable improvements in the quality, naturalness, and adaptability of TTS systems for speech synthesis.

In the 2020s, AI TTS underwent groundbreaking changes, introducing innovations such as voice cloning and zero-shot learning, ultimately achieving an unparalleled level of natural and expressive speech synthesis. These advancements laid the cornerstone for the modern TTS technology we have today.

AI TTS has made an indelible impact on various domains, including business, education, and society, by transcending language barriers and gaining the ability to convey emotions and individuality. However, its widespread adoption has concurrently raised valid concerns regarding privacy and bias.

Historical Context

The history of speech synthesis can be traced back to the 18th century. The first machine that attempted to produce human speech was created by Christian Kratzenstein in 1769. This machine used hollow tubes to resemble resonators and by adjusting resonators’ lengths and shapes, the machine produced five vowels A, E, I, O and U. In 1791, Wolfgang von Kempelen created a “speaking machine” that simulated the human vocal tract and could produce vowels and consonants using a series of bellows, reeds and mechanical components. Later in the 20th century, Bell Labs developed Voder (Voice Operating Demonstrator) , an electrical speech synthesizer that employed bandpass electronic filters and operator control for sound generation.

The central concept during the early development of speech synthesis was to copy the speech. The approach involved either mechanical or electrical. The former aimed to mimic vocal tract movements, while the latter worked on filtering electrical signals to emulate the vocal tract's function. It is worth noting that these early machines were not text to speech systems, as they convert plain text into speech. Nevertheless, they represented significant milestones in the evolution of modern TTS systems, despite their limitations in producing a full range of sounds or sentences. A video filmed in 1939 about Voder demonstrated that the Voder could only produce a single sentence ‘she saw me’ with varying word stress.

Full-fledged text to speech systems emerged until the late 20th century. Most of TTS systems at that time were rule-based, meaning they generated speech using predefined linguistic and phonetic rules. One of the earliest full text to speech systems MITalk, created by researchers at the Massachusetts Institute of Technology (MIT) in 1960s, was one such rule-based system. It allowed users to hear speech as they typed. However, the speech output still lacked naturalness and expressiveness. Additionally, compiling linguistic rules for different languages requires lots of manual work and due to the flexibility of languages, it is impossible to cover all variations and exceptions.  

From late 20th onwards, new TTS techniques such as Parametric TTS, Hidden Markov Models (HMMs), and Concatenative TTS were developed to address naturalness, adaptability and flexibility issues, greatly improving the performance of TTS. However, the limitations of these techniques are inevitable. For example, despite the naturalness of concatenative TTS, it lacks the ability to generate new voices because it relies heavily on recorded speech data.

Thanks to the advancement of artificial intelligence (AI), recent TTS systems can generate natural, human-like speech by adopting innovative training and learning models, as well as enhanced traditional techniques. In the upcoming section, we will delve deeper into AI techniques.

Key Innovations

The 2020s have marked a significant decade in the evolution of Text-to-Speech (TTS) technology driven by artificial intelligence (AI). This period has witnessed a host of groundbreaking innovations that have further refined and expanded the capabilities of TTS systems. Some key innovations in the 2020s include:

1. Transfer Learning and Pretrained Models

One of the pivotal advancements in the 2020s has been the widespread adoption of transfer learning in TTS. Transfer learning allows knowledge learned from one or more base models to be transferred to other tasks. For TTS, this can include knowledge learned from a general speech model and then applied to personalized voice synthesis. This helps expedite the training of personalized voice models as the model already possesses some general speech characteristics. Models like GPT-3 and BERT, initially designed for natural language processing, have been adapted for TTS tasks. This approach has led to more efficient training and improved performance in TTS systems, with less need for extensive domain-specific data.

2. Rule-based TTS Systems

Rule-based TTS systems have continued to contribute to the advancement of AI-driven TTS since 2020. While the fact that rule-based TTS still faces challenges in terms of naturalness and emotional expressiveness compared to neural TTS systems, rule-based TTS finds its value in specialized fields like medicine and law, where precise pronunciation of domain-specific terms is vital. These systems remain relevant for languages with limited linguistic resources, making them suitable for low-resource languages. Additionally, some TTS systems adopt hybrid approaches, combining rule-based and neural network-based techniques to leverage customization while benefiting from naturalness. They excel in applications requiring high control over speech output, such as accessibility solutions.

3. Concatenative TTS

In the AI era, Concatenative TTS systems have seen new developments aimed at improving their performance and adaptability. These developments include hybrid approaches that combine Concatenative TTS with neural network-driven TTS for higher quality and more natural synthesized speech. Additionally, there is a growing trend toward larger speech databases, enabling better voice selection for smoother and more natural speech synthesis across various text contexts. Real-time applications have seen improvements in reducing latency, making Concatenative TTS more practical for real-time communication, voice assistants, and automated voice responses. Personalized TTS, which leverages AI, allows users to customize synthesized voices to their preferences, with potential applications in education, entertainment, and assistive technologies. Moreover, Concatenative TTS systems are extending their support to multiple languages and dialects, making them applicable to diverse global markets. They also find increasing use in specialized fields such as medicine, law, and science to ensure accurate pronunciation of domain-specific terms.

4. Prosody Modeling

Focusing on prosody, or the melody and rhythm of speech, prosody modeling has been a significant area of advancement. Research in modeling prosody has led to more Prosody and expressiveness enhancement by focusing on refining the prosody and expressiveness of synthesized speech. Advanced models now incorporate prosody-aware training, enabling TTS systems to convey emotions, nuances, and variations in pitch and rhythm more effectively, making the speech sound more natural and human-like.

5. Zero-shot Learning

TTS systems have significantly advanced in recent years with deep learning approaches. These advances have motivated research that aims to synthesize speech into the voice of a target speaker using just a few seconds of speech. This approach is called zero-shot multi-speaker TTS. Innovations in zero-shot learning have allowed TTS models to generate speech in languages and styles they were not explicitly trained on. These models leverage multilingual and cross-lingual capabilities, enabling TTS systems to be more versatile and adaptable to diverse linguistic contexts.

6. Voice Cloning

Voice cloning models are trained to capture the specific speaker's pitch, tone, and speech characteristics, making the generated speech more similar to the specific speaker. This can be achieved using deep learning techniques, such as Generative Adversarial Networks (GANs). With this approach, we saw more advancements in creating personalized and customizable voices since the 2020s. TTS systems can now mimic specific voices or allow users to tailor the characteristics of the generated speech, fostering more engaging and adaptive human-computer interactions.

7. Evolving Neural Architectures and End-to-End Approaches

In recent years, the field of AI-driven TTS has seen remarkable progress. Neural network architectures, particularly Transformers and their variants, have revolutionized TTS research by enhancing parallelization, enabling real-time, and high-quality TTS. These architectures, featuring attention mechanisms and positional embeddings, have become standard for capturing context and improving synthesis quality. In this context, End-to-End Approaches have made significant strides since 2020, streamlining the TTS process by utilizing powerful neural networks to directly transform text into speech waveforms. This development has yielded voices that are more human-like and of higher quality, with enhanced customization and personalization capabilities. However, challenges such as data availability and fine-tuning for less common languages persist, yet End-to-End TTS continues to find application in real-time scenarios and personalized voice synthesis.

Impacts

The advancements in AI Text-to-Speech (TTS) technology in the 2020s have had profound impacts across various domains:

1. Business

Cost-Effective Marketing

AI TTS has allowed businesses to create cost-effective marketing materials by generating high-quality voiceovers for advertisements, promotional videos, and e-commerce product descriptions. This has enabled smaller businesses to compete with larger counterparts.

Elevated Customer Engagement
AI TTS is being used in customer service and support chatbots, providing a more engaging and interactive experience for customers. This technology reduces the need for human operators in routine tasks and enables 24/7 support and quick responses to customer queries.

Multilingual Communication
Companies have expanded their global reach by using AI TTS to provide content in multiple languages, which is especially important for businesses with international customers and markets.

Enhanced Brand Recognition
Customized brand voices can help businesses stand out in a crowded market. With AI TTS, businesses can maintain a consistent brand voice across various touchpoints.

2. Education

Accessibility and Inclusion
TTS technology is being used in education and e-learning platforms to provide audio versions of text content. This benefits students with diverse learning styles and those with reading difficulties.

Language Learning
TTS technology remains an asset in language learning, helping learners improve pronunciation, fluency, and comprehension in various languages.

Personalized Learning
Educational institutions use AI TTS to provide personalized learning experiences, adapting content to individual student needs and preferences.

Teacher Assistance
TTS tools support educators in creating and delivering content, from generating voiceovers for instructional videos to offering speech feedback on assignments.

3. Society

Language Preservation
Cross-lingual text-to-speech (CTTS) has facilitated communication across language barriers and has played a role in preserving and revitalizing low-resource and endangered languages, promoting linguistic diversity and aiding in documentation and communication. This is invaluable in a globalized world, allowing for better understanding and cooperation.

Digital Inclusion
TTS technology promotes digital inclusion by making digital content accessible to individuals with low literacy skills and those with disabilities. Improved TTS technology has greatly enhanced accessibility for individuals with visual impairments. It allows text-based information to be converted into speech, making digital content more accessible to a wider audience.

Entertainment and Content Creation
The entertainment industry has benefited from TTS technology through voice cloning and dubbing. It has become easier to dub movies, create voiceovers, and even bring back historical voices for documentaries and other media productions. AI TTS continues to support voiceovers in video games, audiobooks, and other audio content, contributing to the entertainment experience.

Emergency Communication
During emergencies and crisis situations, AI TTS is used to disseminate critical information rapidly, ensuring public safety and information access.

4. Privacy and Ethical Concerns

Deepfake Threat
The potential for AI TTS to be used in deepfake audio and video content has become a growing concern. This emphasizes the need for robust authentication and content verification mechanisms.

Data Privacy
The collection and storage of voice data for TTS models raise concerns about data privacy. Regulations and guidelines have been developed to address these issues.

Bias and Cultural Sensitivity
The challenge of mitigating bias and ensuring cultural sensitivity in TTS models remains a critical consideration in their development and deployment.

Future Research

1. High-quality Speech Synthesis

The most important goal of TTS is to synthesize high-quality speech. The quality of speech is determined by many aspects that influence the perception of speech, including intelligibility, naturalness, expressiveness, prosody, emotion, style, robustness, controllability, etc. While neural approaches have significantly improved the quality of synthesized speech, there is still large room to make further improvements.

1.1 Affective Speech Synthesis

a. Emotional Vocal Bursts

Within the realm of emotional speech synthesis, a particularly intriguing area of exploration could revolve around emotional vocal outbursts. In the now famous promotional video for Google Assistant, the crowd erupted in cheers as the assistant assured the hairdresser that “taking one second” to look for an appointment was fine with a mere “Mm-hmm.” This example vividly demonstrates the significance of vocal outbursts in conveying emotional reactions. In fact, the synthesis of such vocal bursts was already the focal point of the 2020 ExVo challenge. The most successful approach employed in this challenge, utilizing StyleGAN2, had already yielded promising outcomes, underscoring the considerable potential inherent in this avenue of research.

b. Endowing the Agent with an Artificial Personality

This area has been pursued for several decades. However, this topic has been recently revived in the context of big language models, which can be adapted to emulate a specific personality. As personality has been also shown to manifest in speech signals, it is an evident next step to introduce it to conversational agents as well. In general, as exemplified by the tasks featured in the Computational Paralinguistics Challenge, there exist a plethora of speaker states and traits, which can be modeled from the speech: deception, sincerity, nativeness, cognitive load, likability, interest, and others are all variables that could be added to the capabilities of affective agents.

c. Personalization

Personalization is expected to be another major aspect of future speech synthesis systems. Both the expression and the perception of emotion show individualistic effects, which are currently underexploited in the speech synthesis field. Future approaches can benefit a lot from adopting a similar mindset and adapt the production of emotional speech to a style that fits both the speaker and the listener. Such an interpersonal adaptation effect is also seen in human conversations and is a necessary step to foster communication.

Specifically, child-speech-synthesis is one promising research area. Due to the difficulties in collecting children speech data and understanding children speech, synthesizing children speech has always been challenging. In recent years, neural-network-based TTS systems have been gaining popularity. For instance, Hasija, Kadyan, and Guleria used Tacotron for the development of children’s synthetic speech.  However, the problem of lacking data for children's speech still persists. For future developments, researchers need to define better acoustic features for children's speech. Moreover, pronunciation modelling is required.

d. Interaction between AI and Human

The interactions can be accordingly classified as “successful” or not, depending on the goals of the agent. Coupled with effective speech recognition capabilities, these interactions constitute a natural reward signal, which can be further utilized by their agent to improve their speech synthesis and speech recognition capacities in a lifelong reinforcement learning setup, which still remains an elusive goal for the field of affective computing. An overture to this exciting domain can already be found in intelligent dialog generation, where reinforcement learning is already being used to adjust the linguistic style of an agent or to learn backchanneling responses. This paradigm is expected to be more widely used in TTS in the near future.

1.2 Better Representation Learning

Good representations of text and speech are beneficial for neural TTS models, which can improve the quality of synthesized speech. Some initial explorations on text pre-training indicate that better text representations can indeed improve the speech prosody. How to learn powerful representations for text/phoneme sequence and especially for speech sequence through unsupervised/self-supervised learning and pre-training is challenging and worth further explorations.

2. Efficient Speech Synthesis

It is about how to reduce the cost of speech synthesis including the cost of collecting and labeling training data, training and serving TTS models, etc.

2.1 Data-efficient TTS

Many low-resource languages are lack of training data. How to leverage unsupervised/semi-supervised learning and cross-lingual transfer learning to help the low-resource languages is an interesting direction. For example, the ZeroSpeech Challenge is a good initiative to explore the techniques to learn only from speech, without any text or linguistic knowledge. Besides, in voice adaptation, a target speaker usually has little adaptation data, which is another application scenario for data-efficient TTS.

2.2 Parameter-efficient TTS

Today’s neural TTS systems usually employ large neural networks with tens of millions of parameters to synthesize high-quality speech, which block the applications in mobile, low-end devices due to their limited memory and power consumption. Designing compact and lightweight models with less memory footprints, power consumption and latency are critical for those application scenarios.

2.3 Energy-efficient TTS

Training and serving a high-quality TTS model consume a lot of energy and emit a lot of carbon. Improving energy efficiency, e.g., reducing the FLOPs in TTS training and inference, is important to let more populations to benefit from advanced TTS techniques while reducing carbon emissions to protect our environment.

LLM Review

We first asked GPT to review our text and the prompt we used is: "Act as a reader who has no previous knowledge about AI-driven Text to Speech and provide feedback for this text." The response from GPT are as following:

As a reader with no previous knowledge about AI-driven Text to Speech (TTS), I found this text to be highly informative and well-structured. It provides a comprehensive introduction to the evolution of TTS technology, starting from its historical roots in the 18th century to the groundbreaking innovations of the 2020s.

1. Clarity and Structure: The text is well-organized, with clear section headings, making it easy to follow the progression of TTS technology. The historical context section provides essential background information, and the subsequent sections delve into key innovations, impacts, and future research.
2. Historical Context: The historical context provided in the text is enlightening and helps create a foundation for understanding the evolution of TTS technology. It's fascinating to learn about the early attempts at mimicking human speech.
3. Key Innovations: The section on key innovations is particularly valuable for someone new to the topic. It breaks down the significant advancements in TTS, such as transfer learning, rule-based systems, concatenative TTS, prosody modeling, zero-shot learning, and voice cloning. This helps in understanding the diverse approaches within TTS technology.
4. Impacts: The text effectively highlights the practical applications of AI TTS across various domains, including business, education, and society. It's beneficial to know how this technology is being used in real-world scenarios, such as customer engagement, language preservation, and accessibility.
5. Privacy and Ethical Concerns: Addressing the concerns related to privacy and ethical considerations is crucial. It's good to see that the text acknowledges these issues, particularly the potential misuse of AI TTS in deepfake content.
6. Future Research: The section on future research provides a glimpse into the ongoing developments and potential directions in the field of TTS. It's forward-looking and showcases the evolving nature of the technology.

While we appreciated the positive feedback, we believed that GPT's responses are excessively complimentary and somewhat generic. We then changed the prompt: "Please provide a critique of this popular science article from the perspective of an expert in the TTS (Text-to-Speech) field, highlighting the areas that require improvement." The response from GPT are as following:

This popular science article provides a comprehensive overview of the evolution of Text-to-Speech (TTS) technology, including historical context, key innovations, impacts, and future research directions. However, from an expert perspective in the TTS field, there are some areas that could be improved:

1. In-Depth Technical Insights: While the article covers a wide range of TTS topics, it lacks in-depth technical insights. As a TTS expert, I would be interested in more technical details about the various TTS techniques and models mentioned, such as neural TTS architectures, concatenative TTS improvements, and prosody modeling. Adding more technical depth would make the article more valuable to experts in the field.
2. Voice Cloning and Deepfakes: Given the rise of voice cloning and deepfake technologies, it's crucial to address these topics in the article, discussing both the positive and potentially harmful applications of TTS in these areas.
3. Expand on Impact: While you've touched upon the impacts of AI TTS in various domains, you can provide more specific examples or case studies to illustrate how businesses, educational institutions, and other sectors have benefited from this technology. Real-world examples can make the impacts more tangible to readers.
4. Future Research Directions: The section on future research directions is comprehensive but could be more focused. TTS experts would appreciate a more concise and prioritized list of the most pressing research areas and challenges in the field.
5. Visual Aids: The inclusion of visuals like diagrams or charts to illustrate key concepts and the evolution of TTS techniques would enhance the article's clarity and appeal to a broader audience.
6. Transition Sentences: Use clear transition sentences to guide readers from one section to the next. This can make the text flow more smoothly and aid in comprehension.
7. Shorten Sentences and Paragraphs: Some sentences and paragraphs are quite long. Breaking them up or simplifying complex sentences can improve readability.

This time, GPT gave more professional and targeted suggestions, but we thought that several of them maybe too professional and did not take into account the general audience that reads this wiki page, and GPT apprently ignored the keyword in the prompt "popular science article". So we have ignored suggestions #1, #3, and #5, and adopted suggestions #4, #7 and #8 that help to make the article more readable and comprehensible. We also ignored suggestion #6 because it's difficult to implement (editing pages can't directly insert images).

Team Members

Yilan Wei

Xueying Liu

Xinyi Ma

Jingsi Huang

Wansu Zhu