Festival Speech Synthesis System (1997)

Introduction

Festival offers a general framework for building speech synthesis systems as well as including examples of various modules, which is originally developed by Alan W. Black, Paul Taylor and Richard Caley at the Centre for Speech Technology Research (CSTR) at the University of Edinburgh. As a whole it offers full text to speech through a number APIs: from shell level, though a Scheme command interpreter, as a C++ library, from Java, and an Emacs interface. Festival is multi-lingual (currently English (British and American), and Spanish) though English is the most advanced. Other groups release new languages for the system. And full tools and documentation for build new voices are available through Carnegie Mellon's FestVox project.

The system is written in C++ and uses the Edinburgh Speech Tools Library for low level architecture and has a Scheme (SIOD) based command interpreter for control. Documentation is given in the FSF texinfo format which can generate, a printed manual, info files and HTML.

Festival is free software. Festival and the speech tools are distributed under an X11-type licence allowing unrestricted commercial and non-commercial use alike.

Historical Context

Prior to the 1980s, early synthesis systems commonly employed a string re-writing mechanism as their primary data structure. This mechanism stored linguistic representations as strings and involved rewriting these strings with extra symbols during processing. The main drawback of this approach was the merging of diverse elements, such as words, phrase symbols, stress symbols, and phonemes, into a single string. This approach proved clumsy because it necessitated parsing the string each time a module was called.

Between the 1980s and the mid-1990s, with advancements in programming, Multi-Level Data Structures (MLDS) were introduced into speech synthesis, notably in systems like Delta. MLDS organized linguistic information into separate streams, typically linear lists or arrays of linguistic elements. While it provided some structural organization, it faced limitations in representing non-linear structures, making the handling of tree-like structures challenging. Additionally, as new streams were added, the task of establishing connections to ensure full connectivity became increasingly complex.

In 1996, the development of the Festival Speech Synthesis System commenced, and it quickly emerged as a versatile tool for creating new voices. The Festival system introduced a departure from linear lists by allowing for graph structures, offering greater efficiency in representation. Furthermore, items could be included in different modules, enhancing the system's flexibility. Different modules worked in harmony to produce synthetic speech. This enabled the handling of complex relations and the efficient calculation of information on-the-fly.

Since the 21st century began, Festival has continued its rapid development. Its open-source nature has made it a magnet for researchers and developers worldwide. They have made significant contributions to the system by introducing various speech synthesis engines and language models, thus enabling support for multiple languages and a wide array of application domains.

Key Innovations

Based on the historical context that mentioned above, we have acquired a basic understanding of the early speech synthesis systems as well as its limitations. Over the years, there has been a transition from a String Based Processing System to Systems based on MLDS. This has revealed a comprehensive picture about how Festival stood out from other contemporary speech synthesis systems. While the Delta System, which was based on the MLDS, had gain much reputation in this domain, its short comings were also noteworthy. The linear structure applied by the multi-layered systems can pose constrains on data management, as well as restrict the flexibility of data representation and ability to adapt to complex linguistic relationships and structures.

Flexibility of Multiple Structure

In contrast, Festival diverges significantly from MLDS-based systems in several aspects, which also emerged as key innovations for it. First of all, Festival liberates linguistic items form the constraints of linear lists, making the utilization of various graph structures happen, like trees, lists and other linguistic structure. This intersecting relations also contributes to saving on redundancy.

Furthermore, items within Festival can exist within multiple structures, leading to more efficient and adaptable representations. Significantly, Festival allows for the incorporation of elements in various structures, fostering the creation of more effective and flexible representations. This innovative strategy empowers Festival to overcome the constraints imposed by the linear framework of traditional MLDS-based systems, fundamentally transforming the field of linguistic data processing.

New Method for Concatenate Synthesis

In terms of concatenate synthesis, except for the conventional approach of using a single-instance diphone-based method with an inventory comprising one recording of each diphone type, Festival's speech synthesis system introduced a novel "clunits" technique. This method involves an inventory of units recorded within natural sentences and employs a limited form of unit selection.

To be more specific, the fundamental strategy involves organizing units within a specific category, like a particular sound, based on considerations about the context of pronunciation and intonation. This categorization relies on questions related to linguistic elements, such as whether the unit appears at the end of a phrase or is stressed within a syllable. Then the Festival system constructs decision trees for each sound in the database, with each leaf representing a list of database units determined by the questions leading to that specific leaf. During the synthesis process, Festival utilizes the appropriate decision tree to identify the optimal cluster of potential units for each targeted specification. A search is then conducted to determine the most suitable path through the potential units, taking into account the distance of each unit from its cluster center and the cost associated with joining adjacent units.

This new technique can avoid the issue of estimating weights in a feature-based target distance measure while maintaining sensitivity to prosodic and phonetic distinctions. Additionally, it effectively manages variability in unit sparseness, with the tree-building algorithm initiating a cluster split only when significant variations are identifiable.

Multilingual Compatibility

Another innovation point of this system lies in its multilingual compatibility. Festival supports various languages and can also be adapted to different linguistic contexts. Festival comes with support for English (British and American accent), Welsh and Spanish. Voice packages are available for various other languages, including Castellano, Czech, Finnish, Hindi, Italian, Marathi, Polish, Russian, and Telugu.

Impact

Festival Speech Synthesis' Impacts on Speech Technology

Before the born of festival speech synthesis, languages were often stored as strings, phrase symbols, accents and other elements mixed together, which meant that the string needed to be parsed every time the module was called, which was very clumsy. The current system has given up on string-based processing and uses more MLDS, such as Delta. Its advantage is that it forms a word stream and a syllable stream, which are relatively fixed. However, MLDS also has some disadvantages. For example, when the number of streams is large, every time a new stream is created, all the previous ones need to be linked to one place, which leads to a waste of time. The main impact of Festival speech synthesis on speech synthesis is: first, it allows language structures such as trees and lists to exist in one or more projects at the same time. Second, any amount and type of information is allowed, and there is no need to recompile the information to be stored, which reduces a lot of trouble and saves time. Third, it is more flexible in terms of programming language such as C++ and scheme are both acceptable, and JAVA may also be opened in the future, so programmers have greater flexibility in deciding the use of computer languages.

There is a method called unit selection algorithm in the festival language synthesis system, which is used for the problem of unnatural diphthongs in signal processing. It's a smart way to ensure that the voices we create with Festival sound as natural and pleasant as possible, although rare diphone need to be handled in most cases. It requires less signal processing than standard diphone synthesis, or ideally no signal processing. In order to synthesize a new utterance, the target utterance includes words, syllables, duration, etc., and is usually first made using a statistical model. Subsequently, sequences of units are searched and obtained from different locations in the database to match diphthongs, helping to generate more natural and smooth high-quality audio. First, the Unit selection algorithm allows the system to select the most appropriate speech unit to match the text to be synthesized. This makes synthesized speech more natural because it captures the variation and continuity of human speech rather than simply splicing phonemes together. Secondly, because unit selection has a larger selection space, it is usually able to produce higher-quality synthetic speech. This means that speech synthesis systems using unit selection algorithms can generate sounds that are closer to real human pronunciation. Also important is that it can enhance contextual coherence. Unit selection can better maintain contextual coherence in speech synthesis because it takes into account the matching of surrounding units. This helps eliminate choppy or stilted speech synthesis. More detailed tutorial can be found here.

Festival Speech Synthesis' Impacts on Real Life

In terms of multi-modal speech, Festival speech synthesis’s speech synthesis technology can complement XR devices, allowing users to have voice conversations with virtual objects in virtual or augmented reality environments. For example, in VR games, users can use voice to interact with virtual characters to increase the immersion of the game. Or it has higher efficiency and more realistic scenes in VR driving practice and airplane practice.

Festival speech synthesis's situational awareness technology can also be used in car navigation systems to automatically adjust the speed and clarity of voice instructions based on the vehicle's speed, location and traffic conditions. For example, on highways, voice instructions can be faster, while in urban congestion, the voice can be slowed down to ensure better navigation guidance and take care of the driver's emotions.

Voice assistants such as Siri, Google Assistant and Alexa use Festival's speech synthesis technology to provide users with a voice interface. Users can naturally have verbal conversations with these assistants, ask questions, send instructions and receive vocal responses, giving users a more intuitive experience. Meanwhile, the voice assistant allows users to ask questions and get real-time answers through voice. For example, users can ask for information about the weather, news, etc. and get timely responses. Users can also use the voice assistant to set reminders, calendar events and task lists. The assistant generates voice reminders, allowing users to better organize their daily lives.

Future research

In this section, we propose a direction for future research. The festival speech synthesis system is a powerful and versatile open-source text-to-speech (TTS) system. Although this system supports multiple languages, there may be a limited selection of pre-built voices available for certain languages and dialects and creating new voices can be a time-consuming process.

Since the 2000s, the development of, advancements in Neural Network-Based TTS has contributed greatly to reducing the time it takes to synthesize speech, increasing accuracy, and effective speech collection. HMM-based speech synthesis offers the capability to produce speech in various speaking styles while demanding less storage space.

Since the 2010s then, the market has become more active with the development of commercial TTS-speech synthesis services such as Google, Amazon, and Apple. Speech synthesis services, led by large companies, have contributed to effectively collecting minority-oriented speech for specific languages and dialects and shortening the time it takes to generate new speech after the festival speech synthesis system.

In summary, the festival speech synthesis system in the context of voice synthesis technology is underscored by collecting large data set in specific words. In other words, although pre-built speech data for specific languages and dialects is being collected by large companies, there are still limitations. However, as data on minority languages gradually accumulates due to the development of AI TTS, the field of speech synthesis is expected to develop even more rapidly.

LLM Reviews

We used Chat GPT developed by OpenAI to improve content quality and make up for the shortcomings. We let GPT act as a professor, speech synthesis researcher, the person who wants to jump into this field and the person who doesn’t have a background in speech synthesis. Generally, we got a great feedback from GPT. According to the general feedback from GPT, the content of our wiki page is informative and well-structured, providing a reasonably comprehensive and insightful overview of Festival Speech Synthesis System, including its historical context, key innovations, impacts on the field of speech technology, real-life applications, and future research directions. However, we were mindful that GPT was designed to give relatively good comments. Therefore we were careful to take what we could from it and not what we wouldn't.

The prompts we raised and the corresponding improvement approaches provided by GPT are as follows:

1. Act as an voice technology professor and review this website, grade from 1-100 : 92
   • We let chat gpt as a professor to expect what grade we are getting and want to get a feedback from the person who has a professional background. We got a generally good feedback and several points for improvement from chat gpt as it always is. One of the improvement point we got is "You present a direction for future research, acknowledging the need for more voices in various languages and dialects. However, the section could benefit from more details on the specific areas of potential research". We respect the opinion that this needs to be improved, but we agree with the point that this may be off topic, so we did not change the contents.
2. If you were a speech synthesis researcher, please evaluate the wiki page in terms of speech synthesis, and give us the feedback, which part can we improve?
   • The reason for the GPT review from the standpoint of a speech synthesis specialist is to ensure precise terminology, thorough historical context, and comprehensive coverage of technological advancements in the field.
   • Language Clarity: To ensure accessibility for a broader audience, consider simplifying complex technical terms and concepts, and provide explanations where necessary. We agree on this improvement point, thus we reviewed the whole content and tried to use a clear and concise language in order to enhance the overall comprehensibility of the content.
   • Inclusion of Recent Developments: Continuously update the page to include recent advancements in speech synthesis technology, such as the integration of neural network-based TTS and the latest developments in the commercial TTS services sector. Providing a section dedicated to recent developments can help readers grasp the current state and trends of the field more effectively.
3. If you have some voice background and want to jump into the speech synthesis, does it help you? Any suggestions?
   • We let GPT be as students like us, who already have had rough knowledge about speech synthesis, and try to find out whether this wiki page is helpful and provide suggestions.
   • Technical Details: Adding more technical details about the inner workings of Festival Speech Synthesis, including the algorithms and models it employs, would be beneficial. This would help students grasp the core technology behind this system. According to the feedback from GPT, we add more technical details and information in key innovation part.
   • Demonstrations: Including practical demonstrations or examples of how Festival is used for text-to-speech conversion would be valuable. This could involve audio samples or step-by-step walkthroughs of the process. Challenges and Solutions: Discussing specific challenges in speech synthesis and how Festival addresses them would provide a deeper insight. For example, addressing the issue of diphthongs using the unit selection algorithm.
4. If you didn’t know about speech synthesis before, will you have a comprehensive understanding after reading this wiki page? Which part do you think needs to be explained more clearly and understandably?
   • The reason for the GPT review from the viewpoint of a non-professional is to make sure this wiki page can assist individuals interested in voice technology, especially those with limited prior knowledge, in acquiring knowledge more effectively.
   • Enhancing Clarity with Examples: Incorporating additional real-world illustrations and simplifying technical descriptions can enhance the content's accessibility to a wider audience, particularly those who lack familiarity with speech synthesis and related technologies. In response to this feedback, we have heeded the suggestion and enriched the "Impacts" section with more detailed explanations for improved comprehension.

Team Members

Introduction - Page

Historical context - Jingxuan

Impact - Weihao

Innovation - Ting

Future research - Soogyeong

LLM reviews- together