Advancements in AI TTS (2020s)

Introduction

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean odio turpis, sodales a urna a, rutrum elementum libero. Phasellus pellentesque dapibus odio quis sodales. Duis a dignissim odio. Maecenas lobortis sapien purus, eu laoreet magna varius et. Vestibulum euismod pharetra lorem, id dignissim lorem porta ac. Proin euismod vehicula eleifend. Nunc hendrerit efficitur dolor vitae sodales. Fusce sit amet quam laoreet, aliquet diam at, sollicitudin turpis. Vestibulum eget posuere nibh, sit amet sodales purus. Curabitur vel vulputate eros. Vivamus pellentesque libero non magna iaculis tempor. Aenean sodales velit ut nulla aliquam, ut varius orci blandit. Aliquam semper neque ac rutrum porta.

Historical Context

The history of speech synthesis can be traced back to the 18th century. The first machine that attempted to produce human speech was created by Christian Kratzenstein in 1769. This machine used hollow tubes to resemble resonators and by adjusting resonators’ lengths and shapes, the machine could produce five vowels A, E, I, O and U. In 1791, Wolfgang von Kempelen created a tool known as a “speaking machine”. With a series of bellows, reeds and mechanical components, this complex machine represented the model of human vocal tract and could produce vowels and consonants. Then in the 20th century, Bell lab developed Voder (Voice Operating Demonstrator) , an electrical speech synthesizer whose source signals passed through bandpass electronic filters and the output was controlled by an operator.

As we can tell, the main idea of generating speech during the early development of speech synthesis is to copy the speech (Klatt, 1987). The approach to fulfill this idea is either mechanical or electrical. The former creates machines to mimic movements of the vocal tract, and the latter works on filtering electrical signals, as the vocal tract did to source signals. We should notice that these early machines are by standards not text to speech systems, as they do not generate speech from plain text. However, they are undoubtedly milestones in development of modern TTS systems, and their limitations are also obvious. Machines are not capable of producing a full range of vowels and consonants, let alone sentences. In a video about Voder, it can only produce one sentence ‘she saw me’ with stress on different words.

Until the late 20th century, full-fledged text to speech systems emerged. Most of TTS systems at that time were rule-based, meaning speech is generated using predefined linguistic and phonetic rules, letter to sound rules, and stress to prosody rules. One of the earliest full text to speech systems MITalk, created by researchers at the Massachusetts Institute of Technology (MIT) in 1960s, was also rule-based. The performance of rule-based machines is so much better that users could hear the output speech as they type sentences on the keyboard. However, the speech output still lacks naturalness and expressiveness. What’s more, compiling linguistic rules for languages requires lots of manual work and due to the flexibility of languages, it is not impossible to cover all variations and exceptions.  

From late 20th onwards, more TTS techniques have been developed such as Parametric TTS, Hidden Markov Models (HMMs) , and Concatenative TTS. They dealt with the naturalness, adaptability and flexibility problems to some extent and improved the performance of TTS systems. But they all have their downsides. For example, despite the naturalness of concatenative TTS, it lacks the ability to generate new voices because it relies heavily on recorded speech data.

The recent TTS system, on the contrary, can generate natural and human-like speech thanks to entwinement with artificial intelligence. They have been improved in multiple ways including training models and techniques, functionalities and output performance. We will look deeper into the AI techniques in the field of TTS in the next section.

Key Innovations

The Voder was among the first devices to allow manual control of speech synthesis. It was a pioneer in electronic sound generation, breaking down human speech into its fundamental acoustic components and reproducing these patterns electronically: this was a significant advancement in the early stages of electronic speech synthesis. Moreover, Voder was the first successful attempt at recreating an important physiological characteristic of the human voice – the ability to create voiced and unvoiced sounds.

To improve the operator's performance, the Voder had a recording and playback feature that allowed operators to objectively analyze their areas of improvement. This feature is similar to modern-day contact centers that use call recording and analysis to improve agent performance.

Impact

The advancements in AI Text-to-Speech (TTS) technology in the 2020s have had profound impacts across various domains:

Business

Cost-Effective Marketing

AI TTS has allowed businesses to create cost-effective marketing materials by generating high-quality voiceovers for advertisements, promotional videos, and e-commerce product descriptions. This has enabled smaller businesses to compete with larger counterparts.

Elevated Customer Engagement

AI TTS is being used in customer service and support chatbots, providing a more engaging and interactive experience for customers. This technology reduces the need for human operators in routine tasks and enables 24/7 support and quick responses to customer queries.

Multilingual Communication

Companies have expanded their global reach by using AI TTS to provide content in multiple languages, which is especially important for businesses with international customers and markets.

Enhanced Brand Recognition:

Customized brand voices can help businesses stand out in a crowded market. With AI TTS, businesses can maintain a consistent brand voice across various touchpoints.

Education

Accessibility and Inclusion

TTS technology is being used in education and e-learning platforms to provide audio versions of text content. This benefits students with diverse learning styles and those with reading difficulties.

Language Learning

TTS technology remains an asset in language learning, helping learners improve pronunciation, fluency, and comprehension in various languages.

Personalized Learning

Educational institutions use AI TTS to provide personalized learning experiences, adapting content to individual student needs and preferences.

Teacher Assistance

TTS tools support educators in creating and delivering content, from generating voiceovers for instructional videos to offering speech feedback on assignments.

Society

Language Preservation

Cross-lingual text-to-speech (CTTS) has facilitated communication across language barriers and has played a role in preserving and revitalizing low-resource and endangered languages, promoting linguistic diversity and aiding in documentation and communication. This is invaluable in a globalized world, allowing for better understanding and cooperation.

Digital Inclusion

TTS technology promotes digital inclusion by making digital content accessible to individuals with low literacy skills and those with disabilities. Improved TTS technology has greatly enhanced accessibility for individuals with visual impairments. It allows text-based information to be converted into speech, making digital content more accessible to a wider audience.

Entertainment and Content Creation

The entertainment industry has benefited from TTS technology through voice cloning and dubbing. It has become easier to dub movies, create voiceovers, and even bring back historical voices for documentaries and other media productions. AI TTS continues to support voiceovers in video games, audiobooks, and other audio content, contributing to the entertainment experience.

Emergency Communication

During emergencies and crisis situations, AI TTS is used to disseminate critical information rapidly, ensuring public safety and information access.

Privacy and Ethical Concerns

Deepfake Threat

The potential for AI TTS to be used in deepfake audio and video content has become a growing concern. This emphasizes the need for robust authentication and content verification mechanisms.

Data Privacy

The collection and storage of voice data for TTS models raise concerns about data privacy. Regulations and guidelines have been developed to address these issues.

Bias and Cultural Sensitivity

The challenge of mitigating bias and ensuring cultural sensitivity in TTS models remains a critical consideration in their development and deployment.

Future Research

High-quality speech synthesis

The most important goal of TTS is to synthesize high-quality speech. The quality of speech is determined by many aspects that influence the perception of speech, including intelligibility, naturalness, expressiveness, prosody, emotion, style, robustness, controllability, etc. While neural approaches have significantly improved the quality of synthesized speech, there is still large room to make further improvements.

Affective speech synthesis

a. Emotional vocal bursts

Within the realm of emotional speech synthesis, a particularly intriguing area of exploration could revolve around emotional vocal outbursts. In the now famous promotional video for Google Assistant, the crowd erupted in cheers as the assistant assured the hairdresser that “taking one second” to look for an appointment was fine with a mere “Mm-hmm.” This example vividly demonstrates the significance of vocal outbursts in conveying emotional reactions. In fact, the synthesis of such vocal bursts was already the focal point of the 2020 ExVo challenge. The most successful approach employed in this challenge, utilizing StyleGAN2, had already yielded promising outcomes, underscoring the considerable potential inherent in this avenue of research.

b. Endowing the agent with an artificial personality

This area has been pursued for several decades. However, this topic has been recently revived in the context of big language models, which can be adapted to emulate a specific personality. As personality has been also shown to manifest in speech signals, it is an evident next step to introduce it to conversational agents as well. In general, as exemplified by the tasks featured in the Computational Paralinguistics Challenge, there exist a plethora of speaker states and traits, which can be modeled from the speech: deception, sincerity, nativeness, cognitive load, likability, interest, and others are all variables that could be added to the capabilities of affective agents.

c. Personalization

Personalization is expected to be another major aspect of future speech synthesis systems. Both the expression and the perception of emotion show individualistic effects, which are currently underexploited in the speech synthesis field. Future approaches can benefit a lot from adopting a similar mindset and adapt the production of emotional speech to a style that fits both the speaker and the listener. Such an interpersonal adaptation effect is also seen in human conversations and is a necessary step to foster communication.

Specifically, child-speech-synthesis is one promising research area. Due to the difficulties in collecting children speech data and understanding children speech, synthesizing children speech has always been challenging. In recent years, neural-network-based TTS systems have been gaining popularity. For instance, Hasija, Kadyan, and Guleria used Tacotron for the development of children’s synthetic speech.  However, the problem of lacking data for children's speech still persists. For future developments, researchers need to define better acoustic features for children's speech. Moreover, pronunciation modelling is required.

d. Interaction between AI and human

The interactions can be accordingly classified as “successful” or not, depending on the goals of the agent. Coupled with effective speech recognition capabilities, these interactions constitute a natural reward signal, which can be further utilized by their agent to improve their speech synthesis and speech recognition capacities in a lifelong reinforcement learning setup, which still remains an elusive goal for the field of affective computing. An overture to this exciting domain can already be found in intelligent dialog generation, where reinforcement learning is already being used to adjust the linguistic style of an agent or to learn backchanneling responses. This paradigm is expected to be more widely used in TTS in the near future.

Better representation learning

Good representations of text and speech are beneficial for neural TTS models, which can improve the quality of synthesized speech. Some initial explorations on text pre-training indicate that better text representations can indeed improve the speech prosody. How to learn powerful representations for text/phoneme sequence and especially for speech sequence through unsupervised/self-supervised learning and pre-training is challenging and worth further explorations.

Efficient speech synthesis

It is about how to reduce the cost of speech synthesis including the cost of collecting and labeling training data, training and serving TTS models, etc.

Data-efficient TTS

Many low-resource languages are lack of training data. How to leverage unsupervised/semi-supervised learning and cross-lingual transfer learning to help the low-resource languages is an interesting direction. For example, the ZeroSpeech Challenge is a good initiative to explore the techniques to learn only from speech, without any text or linguistic knowledge. Besides, in voice adaptation, a target speaker usually has little adaptation data, which is another application scenario for data-efficient TTS.

Parameter-efficient TTS

Today’s neural TTS systems usually employ large neural networks with tens of millions of parameters to synthesize high-quality speech, which block the applications in mobile, low-end devices due to their limited memory and power consumption. Designing compact and lightweight models with less memory footprints, power consumption and latency are critical for those application scenarios.

Energy-efficient TTS

Training and serving a high-quality TTS model consume a lot of energy and emit a lot of carbon. Improving energy efficiency, e.g., reducing the FLOPs in TTS training and inference, is important to let more populations to benefit from advanced TTS techniques while reducing carbon emissions to protect our environment.

LLM Review

-

Team Members

xinyi xueying jingsi yilan wansu