Intro to Voice Technology syllabus

Introduction

In this course, we will explore the foundations of speech synthesis and recognition, delving into the interplay between technology and language.

Learning outcomes

Upon the successful completion of the course “Introduction to Voice Technology”, you will be able to:

1. explain the history of voice technology.
2. explain the basic elements of speech synthesis and recognition.
3. identify data resources for voice technology applications and know where to find them.
4. describe data management requirements for collecting and storing speech and speaker data.
5. elaborate on the value and relative importance of data management, licensing and privacy issues concerning speech and speaker data.
6. describe core aspects within speech production and feature extraction.
7. discuss with peers how human factors and relevant aspects of context affect the interaction between humans and voice technology systems.
8. describe how the user acceptance of a voice technology application can be investigated.

Course structure

The course runs for 8 weeks. Each week has 2 classes of 1 hour 45 minutes (with a 15 minute break in the middle).

Classes are on Tuesday and Wednesday, 13:15 -- 15:00.

Guest speakers

The following guest speakers will contribute to this course.

• Dr Laurent Besachier, Principal Scientist and NLP Group Lead at Naver Labs (EU).  Topics of interest
• Dr. Loredana Cerrato, Project Manager Nuance - Microsoft. See blog post.
• Dr Leigh Clark, Senior UX Researcher - Bold Insight UK
• Dr Jide Edu, Security Researcher at the Alan Turing Institute (London, UK)
• Dr Frederic Robinson, Founder of LeapTech (Basel, Switzerland)
• Dr Lorenzo Tarantino, CTO at Voiseed

Practical Information

Literature

We will mostly be reading literature that is available online. Obligatory readings are either accessible through open access or online through SmartCat of the library.

Brightspace

We use the virtual learning environment “Brightspace” as the main platform for communication. If there is any necessary change on the syllabus, I will announce it in class and in Brightspace.

Assessment

Assignment	%
Wiki page 1	20
Wiki page 2	20
Wiki page 3	20
Talking clock	10
Talking clock presentation	10
Participation activities	20
TOTAL	100

Cheating and plagiarism

Cheating and plagiarism are academic offenses, with severe consequences. They are acts or omissions by students to partly or wholly hinder accurate assessment. As per the Teaching and Examination Regulations, cases of cheating and plagiarism are reported by the instructor to the Board of Examiners, which will decide on the consequences.

Contact information

Your instructors for the course are Dr Matt Coler (m.coler@rug.nl) and Dr Joshua Schäuble (j.k.schauble@rug.nl). For general questions or suggestions you can contact the Educational Secretary or Student Service Desk (cf-sec@rug.nl, +31(0) 58 205 5009).

Planning

Week 1: Intro to intro

We start the journey with an overview of the whole program and consider the field of voice technology in terms of academic disciplines. You will be able to:

• see the MSc Voice Technology from a broader perspective.
• have a basic idea of speech synthesis and speech recognition
• give an overview of the research field of voice technology

Class I: Getting started  (Sept 5)

Welcome! In this first class we will get to know one another. You will learn about the MSc Voice Tech program, the team of researchers, visiting scholars, and PhDs, hear more about the events and guest lectures scheduled, and acquire an understanding of the final thesis project.

Preparation

• Read the syllabus, and provide your questions and comments here.
• Complete this questionnaire.

Class II: The field (Sept 6)

In this class, we will have a guest lecture by Loredana Cerrato (Nuance) about the history of the field, charting the path from the past to the present.

Preparation

• Watch this video and read this article about speech recognizers and synthesizers. When you're done, check out this popular content about audio recording, speech synthesis, and speech recognition.
  • Optionally, you may also find this text by Thaker & Harvashu interesting: History of the sound recording technology.

• Check out the Activity 1 if you want to get a headstart.

Week 3: Synthesis

Class I: Synthesis for video games and more (Sept 19)

In this class will will start addressing some of the history of speech synthesis. We will also meet Lorenzo Tarantino (CTO, Voiseed, an Italian start-up specializing in synthesis).

Preparation

• Balyan, A. et al. (2013). Speech synthesis: a review. International Journal of Engineering Research & Technology (IJERT), 2(6), 57-75.
• Johnson, Stephen (2023). This MIT scientist gave Stephen Hawking his voice — then lost his own. Big Think. [popular article]
• Watch “Accidentally famous: the originally voice of Siri – TEDx-talk” (2016)

Preparation

• Check out Voiseed's webpage.

Homework

• Assignment 2 [due Monday]

Class II: (Sept 20): SOTA  (Sept 20)

• Guest lecture by Dr Beacier

Preparation Required reading:

1. Besacier, L., Barnard, E., Karpov, K. & Schultz, T. (2014). Automatic speech recognition for under-resourced languages: A survey. Speech Communication.
2. Other material provided by Dr Beachier [tbd]

Optional reading:

• Arora, S. J. & Singh, R. P. (2012). Automatic Speech Recognition: A Review. International Journal of Computer Applications, 60(9):34-44.
• Juang, B. H., & Rabiner, L. R. (2004). Automatic Speech Recognition – A Brief History.
• O’Shaughnessy, D. (2019). Recognition and Processing of Speech Signals Using Neural Networks. Circuits, Systems, and Signal Processing, 38:3454-3481. doi: 10.1007/s00034-019-01081-6

Pass/Fail participation-related assignments

Participation: There are multiple ways to participate in class aside from talking. Therefore participation will be assessed in an inclusive way taking into account your engagement in group/individual activities, your connections with guest speakers, any additional peer review activities, and the way in which you support the class overall. To those ends, I’ll take into account your self-assessment which you will deliver to me via a form.

Activity 1: Exploring ASR Accuracy in Different Environments

Objective: Understand the impact of different environments, conditions, and hardware on speech recognition accuracy without requiring software installation.

Introduction: Speech recognition is everywhere, from voice assistants to transcription services. In this simple activity, you'll explore how speech recognition accuracy changes in various settings without the need for software installation.

Assignment Overview: You'll record your voice in different environments using different hardware setups. Then, you'll use a user-friendly online speech recognition tool to analyze accuracy differences across conditions.

Instructions:

1: Recording Your Voice:

• Environments: Choose three different locations (indoors/outdoors, at a loud cafe, near a busy street, etc.)
• Hardware: Use your smartphone, laptop, or any device with a microphone.
• Record: In each environment, record yourself reading the provided text. Label each recording with the environment and device used. Some inspiration:
  • Coler_iPhoneXR_cafe-normal
  • Coler_iPhoneXR_traffic-whispering
  • Coler_iPhoneXR_forest-yelling
  • Coler_iPhoneXR_bar-speaking-very-quickly
  • Coler_iPhoneXR_plaza-normal-while-running

2a: Beginner's version: Using Google Docs Voice Typing: Go to https://docs.google.com/. Make sure you're signed in to your Google account. Click on the "+ New" button and select "Google Docs"

Enable Voice Typing:

• In the top menu, go to "Tools" > "Voice typing..."
• A microphone icon will appear on the left side of the document.

Upload Your Recordings:

• Open a file explorer and locate the recording you want to transcribe.
• Play the recording on your device (or from your phone directly), and as it plays, click the microphone icon in Google Docs to start voice typing.

Transcription Process:

• Google Docs Voice Typing will start transcribing the audio as it hears it.

Review Transcription:

• The transcription will appear on the document in real-time
• Review the transcription for accuracy as the audio plays.

Note Discrepancies:

• Compare the transcribed text to what you actually said in the recording.
• Note any differences or errors in the transcription.

Stop Voice Typing:

• Click the microphone icon again to stop voice typing once the entire recording is transcribed.

Repeat for Other Recordings:

• Repeat the above steps for each of the recordings you made in different environments and with different hardware setups.

Compile Transcriptions:

• Organize the transcriptions and any notes about accuracy discrepancies for each recording.

Proceed to Analysis:

• With your transcriptions ready, you can move on to Step 3 (Compare Accuracy) and analyze the differences in accuracy across conditions.

2b: Advanced version: Use the SpeechRecognition Python library if you’re more technically proficient. If you're interested indelving into the technical aspects of speech recognition, you have the opportunity to explore the SpeechRecognition Python library. This library provides a programmatic way to interact with speech recognition engines, enabling you to transcribe spoken words into text using code. The SpeechRecognition library is a Python package that offers a range of functionalities for working with speech-to-text conversion. It acts as an interface to several popular speech recognition engines, making it easier for developers to incorporate speech recognition capabilities into their applications.

Install the SpeechRecognition library using pip:

pip3 install SpeechRecognition

Write a Python script that utilizes the library to transcribe your recorded audio files.

Include detailed comments in your code to explain each step of the process, making it accessible for peers who might be new to coding in the Wiki.

Document any challenges you faced and how you overcame them during the transcription process.

3: Compare Accuracy:

• Review Transcriptions: Examine the transcriptions for each recording.
• Note Differences: Compare the transcriptions to what you actually said. Note any discrepancies.

4: Presentation:

• Create demo: Use Slides to create a presentation. Include samples of your recordings, the transcriptions, and a comparison of accuracy.

5: Discussion:

• Bring your presentation and recordings to class.
  • Are there certain types of errors that appear across different environments?
  • How might background noise or variations in speech volume impact accuracy?
  • Can you identify any patterns in accuracy discrepancies based on the hardware used?

What to upload into Brightspace:

• ZIP folder with the recordings, signed consent form, and a readme folder
• Presentation you made in step 5