Editing State-of-the-art (section)

==== Zhao et al. (2018): Accent conversion using phonetic posteriograms ====
'''Summary''': Accent conversion (AC) means transforming non-native speech to sound as if the speaker had a native accent, or vice-versa. The main challenge faced in traditional methods of voice conversion is decoupling the speaker’s voice quality from their pronunciation (i.e. teasing apart accent information and keeping everything else acoustically unchanged). Additionally, when mapping source spectra from a native speaker into the acoustic space of an L2 speaker, previous attempts focus on acoustic similarity: changing formants- and pitch trajectories, blending spectral envelopes. The alternative used here is, in turn, is phonetic similarity, which maps source to target based on an intermediate phonetic label.
The phonetic posteriograms are computed using a DNN-based acoustic model. The distance between these phonetic posterior feature vectors is calculated to find the closest pairs of frames between source (native) and target (L2) speakers. The frame pairs are used to train a GMM. The two baselines used are acoustic similarity matching and dynamic time warping.

Experimental setup: get Kaldi DNN acoustic model, train it on Librispeech data, get native English speech (CMUArctic) and non-native recordings (Hindi, Korean, Arabic), use STRAIGHT for speech decomposition, MFCC extraction, train GMMs (128 components), synthesize speech by reconstructing spectrograms and adding aperiodicity.

'''RQ:''' How can accent-related features be successfully decoupled from speaker-related features, to achieve non-native to native voice conversion while preserving speech quality?

'''Results:''' Synthesized results were compared to baselines through listening tasks using Mechanical Turk (rating acoustic quality, speaker identity y/n, nativeness of resynthesized speech):

* significantly higher acoustic quality ratings compared to baselines.
* comparable speaker identity scores.
* strong preferrence for posteriogram conversions by native EN speakers as more ‘native-like‘ compared to baselines and original L2 utterances.

'''Critical observations:''' This paper addressed the opposite issue, namely converting foreign-accented speech to sound like native one (mainly for educational purposes). This still means you need to figure out which features are related to accent, and which features are related to anything else, but is arguably the easier thing to do, as it requires to drop information instead of successfully adding it. Additionally, the approach is not entirely explainable, because posteriograms are encoder features and it's not always transparent what is learned to be most relevant. Lastly, this approach likely works increasingly worse the fewer speakers there are in a dataset. Even if you accented speech data, one speaker can only have one accent, so in case the number of speakers is small, the model might learn to encode speaker identity instead of accent features.

'''Relevance:''' It is important to know that given enough speakers and enough data, accent features can be decoupled from other speech features and dropeed to obtain a higher perceived 'nativeness' of the speech.