Editing State-of-the-art (section)

=== Synthetically improving foreign-accented speech recognition ===

==== Introduction ====
More often than not, speech corpora either contain only native speech, or the non-native subset is significantly underrepresented. At the same time, gender and foreign accent are the most salient factors contributing to changes in the acoustics of speech. However, not only are there numerous possible combinations of L1 and L2s, but the annotation and labelling os recordings to a suitable degree (e.g. age of L2 acquisition, country of origin, L1, L2 proficiency, language of education etc. are all factors that should be reported in order to make the speech resources reliable and usable) are laborious and expensive.

In light of these challenges, methods of synthetical data augmentation have been recently explored in the literature. While creating synthetically-accented data through accent conversion models (ACMs) is a straightforward, inexpensive, and oof-the-shelf approach, it is not without limitations and the degree to which recognition performance is improved through such approaches depends on several factors. The following three articles provide some insight into these approaches and highlight both major advantages and persistent challenges.

==== Zhao et al. (2018): Accent conversion using phonetic posteriograms ====
'''Summary''': Accent conversion (AC) means transforming non-native speech to sound as if the speaker had a native accent, or vice-versa. The main challenge faced in traditional methods of voice conversion is decoupling the speaker’s voice quality from their pronunciation (i.e. teasing apart accent information and keeping everything else acoustically unchanged). Additionally, when mapping source spectra from a native speaker into the acoustic space of an L2 speaker, previous attempts focus on acoustic similarity: changing formants- and pitch trajectories, blending spectral envelopes. The alternative used here is, in turn, is phonetic similarity, which maps source to target based on an intermediate phonetic label.
The phonetic posteriograms are computed using a DNN-based acoustic model. The distance between these phonetic posterior feature vectors is calculated to find the closest pairs of frames between source (native) and target (L2) speakers. The frame pairs are used to train a GMM. The two baselines used are acoustic similarity matching and dynamic time warping.

Experimental setup: get Kaldi DNN acoustic model, train it on Librispeech data, get native English speech (CMUArctic) and non-native recordings (Hindi, Korean, Arabic), use STRAIGHT for speech decomposition, MFCC extraction, train GMMs (128 components), synthesize speech by reconstructing spectrograms and adding aperiodicity.

'''RQ:''' How can accent-related features be successfully decoupled from speaker-related features, to achieve non-native to native voice conversion while preserving speech quality?

'''Results:''' Synthesized results were compared to baselines through listening tasks using Mechanical Turk (rating acoustic quality, speaker identity y/n, nativeness of resynthesized speech):

* significantly higher acoustic quality ratings compared to baselines.
* comparable speaker identity scores.
* strong preferrence for posteriogram conversions by native EN speakers as more ‘native-like‘ compared to baselines and original L2 utterances.

'''Critical observations:''' This paper addressed the opposite issue, namely converting foreign-accented speech to sound like native one (mainly for educational purposes). This still means you need to figure out which features are related to accent, and which features are related to anything else, but is arguably the easier thing to do, as it requires to drop information instead of successfully adding it. Additionally, the approach is not entirely explainable, because posteriograms are encoder features and it's not always transparent what is learned to be most relevant. Lastly, this approach likely works increasingly worse the fewer speakers there are in a dataset. Even if you accented speech data, one speaker can only have one accent, so in case the number of speakers is small, the model might learn to encode speaker identity instead of accent features.

'''Relevance:''' It is important to know that given enough speakers and enough data, accent features can be decoupled from other speech features and dropeed to obtain a higher perceived 'nativeness' of the speech. 

==== Jin et al. (2023): Voice-preserving zero-shot multiple accent conversion ====
'''Summary:''' Separating accent from speaker identity is usually the hardest, because each speaker in the dataset has one single accent. Previous attempts at doing this include:

* use adversarial learning to get a discriminator to wipe out speaker-dependent information from content embeddings.
* quantization of different features in speech to obscure undesired information.

The main problem with conventional approaches to conversion is that they very often require available utterances with the same text in both source and target accent, making their applicability very limited. Alternatively, different approaches require either training or fine-tuning on the input utterances.

The current paper uses a pronunciation encoder, an acoustic encoder, and a HiFiGAN voice decoder. During training, the model minimises reconstruction loss between input and output mel-spectrograms. The pronunciation encoder synthesizes accent-dependent pronunciation sequences using accent IDs. The acoustic encoder mapss MFCCs and periodicity features to a single vector, while adversarial training removes accent information. Lastly, the decoder reconstructs waveforms from the processed features. The model is evaluated on audio quality, speaker similarity, and accent conversion effectiveness.

'''Results:''' Results indicate it maintains comparable audio quality to the original, maintains speaker similarity, and is efficient in replicating perceived nativeneess. However, listeners struggled to identify synthesized accents if they were unfamiliar with the target language (e.g. a native US listener could not classify a Korean accent on English as such, but a bilingual Korean-American listener could). Overall, the paper presents one of the best performing ACMs, that is able to preserve both speaker identity and acoustic quality during conversion. 

'''Critical observations:''' I think this paper achives a lot given that it's zero shot, but I am a bit critical about just how 'zero-shot' it truly is. They use a pre-trained acoustic model and while they do not require accent labels or speaker IDs, it seems that their training set contains over 24h of accented speech for all accents that they're synthesizing in. Additionally, none of their code is openly available, which is understandable for a private corporation like Meta, but it's still a bit disappointing.

==== Klumpp et al. (2023): Synthetic cross-accent data augmentation for ASR ====
'''Summary:''' Foreign-accentes speech is usually underrepresented in, if not absent from speech corpora. Auxiliary input (learned accent embeddings, intermediate wav2vec2.0 representations) can address the decreased ASR recognition on this type of speech; the challenge remains that of achieving good accent conversion while preserving source speaker voice characteristics. The current approach builds on a pre-existing ACM by Jin et al. (2023) -- see above -- and aims to provide synthetic ASR training data using it. Phonetic knowledge is crucially injected into training to improve accent-specific pronunciation, and learnable accent representations are introduced to allow for variable accent strengths and adaptability to unseen accents.

The experimental setup involved evaluating two ASR models using Librispeech data. The first model (Base) utilized an efficient memory transformer followed by a recurrent neural transducer (RNNT), while the second model (HuBERT) had a similar structure with adjustments in channel configurations and dropout probabilities. The ASR models were tested on Librispeech data and accents from L2-Arctic corpus and Accented Vox Populi (AVP) dataset. 

In experiments, the baseline ASR systems were trained without synthetic accented speech data, then evaluated. Three additional ASR models were trained with a combination of real and synthetic accented data, using a ratio of 80% real and 20% synthetic data. The ratio remained consistent across all accents. Finally, learned accent embeddings from L2-Arctic samples were visualized using t-SNE plots to assess their suitability for encoding accent information in an Accent Conversion Model (ACM).

'''RQ:''' Is it possible to improve ASR of accented speech with synthetic samples of a particular accent?

'''Results:''' The inclusion of one synthetic accent during ASR training had a positive effect on recognition results for that particular accent, which was a clear indicator that the ACM was able to synthesize a sufficient degree of accentedness. At the same time, HuBERT'd performance decreased with the use of synthetic data, likely due to the fact that it was not pre-trained on any and fine-tuning did not do enough. The Base model, which was trained from scratch, had a much grater benefit from the synthetic data. Notably, even when all seven accents were introduced in training, this did not improve performance on other unseen accents.

Overall, including one synthetic accent improved performance on that accent; and including several accents improved performance on those accents, but none of the conditions improved recognition on accents not seen in training. Additionally, pre-trained HuBERT did not benefit much from additional synthetic data fine-tuning, whereas a model trained from scratch saw much greater benefit from this approach.

'''Critical observations:''' Again, none of this replicable because the code is not available. It would have been also interesting to see a bit more ASR models be tested on this; this particular comparison does highlight the pre-trained/trained from scratch distinction in performance on this task, but there are other models that are seemingly good candidates and were not included.

'''Relevance:''' The authors show the potential for using synthetically accented data as a data augmentation approach to improve ASR performance on foreign-accented speech.

==== General insights ====
The synthesis of accented speech as a data augmentation method in ASR is promising for improving recognition performance on non-native speech. The three articles reviewed provide valuable insights into accent conversion methods and their implications for ASR systems. Zhao et al. (2018) shows the effectiveness of phonetic posteriograms in converting foreign-accented speech to sound more native-like and successfully decouples accent-related features from other speech characteristics. Jin et al. (2023) proposed a zero-shot multiple accent conversion approach, maintaining audio quality and speaker identity during conversion, albeit with limitations in accent classification for unfamiliar listeners. Klumpp et al. (2023) extended this work by integrating synthetic accented speech data into ASR training, showing improvements in recognition performance on the trained accents. However, the effectiveness varied depending on the model architecture, with pre-trained models benefiting less from synthetic data than models trained from scratch. Despite promising results, the lack of code availability and limited generalizability to unseen accents pose challenges for broader adoption. Overall, while accent conversion models offer a promising strategy for data augmentation in ASR, further research should focus on generalization and replicability for real-world applications.

==== References ====
Jin, M., Serai, P., Wu, J., Tjandra, A., Manohar, V., & He, Q. (2023, June). Voice-preserving zero-shot multiple accent conversion. In ''ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'' (pp. 1-5). IEEE.

Klumpp, P., Chitkara, P., Sarı, L., Serai, P., Wu, J., Veliche, I. E., ... & He, Q. (2023). Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition. ''arXiv preprint arXiv:2303.00802''.

Zhao, G., Sonsaat, S., Levis, J., Chukharev-Hudilainen, E., & Gutierrez-Osuna, R. (2018, April). Accent conversion using phonetic posteriorgrams. In ''2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'' (pp. 5314-5318). IEEE.