Librispeech: Difference between revisions

From MSc Voice Technology
Jump to navigation Jump to search
No edit summary
 
(5 intermediate revisions by 2 users not shown)
Line 176: Line 176:


=== Analysis ===
=== Analysis ===
In the WER overview, there is only one sentence (0012) are correctly transcribed with the WER in 0. However, it seems that more transcriptions should be considered correct. For example, in 0000, both Google and Whisper transcribe "Mister" as "Mr." and Google transcribe "we are" as "we're", which lead to 5.88 WER and 17.65 WER, respectively. There is actually no difference in meaning between these abbreviations. Same situation happened in 0001 of Whisper transcription where "Mister" has been written as "Mr.", causing 10 WER.
Abbreviations and different spellings cause many misjudgements in calculating WER. For example, in 0000, both Google and Whisper transcribed "Mister" as "Mr." and Google transcribed "we are" as "we're", which lead to 5.88 WER and 17.65 WER respectively. Same situation happened in 0001 and 0013 of Whisper transcriptions where "Mister" has been written as "Mr.". Different spellings, for example in Whisper 0011 where "recognising" has been written as "recognizing", together with "Mr.", leading to 5.88 WER. Since "Mr." and "Mister", "recognizing" and "recognising", "we are" and "we're" have no difference at all except for their spellings, we should consider above-mentioned transcriptions as correct with 0 WER.  


=== Conclusion ===
=== Conclusion ===
Line 297: Line 297:
|by Harry Quilter, M.A.
|by Harry Quilter, M.A.
|}
|}
Recording device: iPhone XR
Environment: relatively quiet corridor with occasional background noise, standard speech rate
Notes:
In general, the WER in this transcription is considerably higher than that of the original audiobook transcribed by Whisper. The primary reason behind this discrepancy lies in the fact that this recording was made by a native Mandarin speaker, and there are instances where the pronunciation of certain phonetics is not entirely accurate when speaking in continuous sentences. For example, words like 'man' and 'men,' which have similar pronunciations, lead to a higher WER.
However, there are a few exceptions to this trend. Notably, the WER for segments 0008 and 0009 in this transcription is lower than that of the original recording. This is especially evident in segment 0009, which features a lengthy sentence with no transcription errors. This indicates that Whisper's Large Vocabulary Continuous Speech Recognition (LVCSR) system performs admirably in specific cases.
There are instances where Whisper faces challenges. Firstly, it struggles when dealing with words that share the same or similar pronunciations. For instance, in segment 0003, the original word 'WHETHER' is transcribed as 'weather.' Another instance occurs in segment 0011, where 'PHASES' is transcribed as 'faces.' This highlights the need for improvement in the system's ability to transcribe within the context of the language model.
Additionally, the transcription quality suffers when it comes to liaison. For example, in segment 0000, the original transcript reads 'apostle,' but the transcription results in 'a parcel.' A similar situation arises in segment 0004, where 'AT EM' is transcribed as 'atom.'


=== M.Tepei ===
=== M.Tepei ===

Latest revision as of 16:16, 28 September 2023

Dataset abstract[edit | edit source]

Dataset name Type of data Annotation remarks License Metadata
LibriSpeech FLAC
  • The audio file and the transcription file are stored under the same directory.
  • There are multiple audio files and only one txt file which is the transcription.
  • And in the transcription file, the transcription for each audio is in separate lines.
Creative Commons Attribution 4.0 International License.
  • Sampling rate: 16 kHz
  • Download size: 57.14 GiB

Comparison Performance between APIs[edit | edit source]

We use a subset(1272/128104) of dataset dev-clean(development set, "clean" speech) to compare the performance between the Google and Whisper APIs.

This is the evaluation script: https://gist.github.com/auwjong22/357c3020a82d68414aae9fdcb3b60521

WER Overview[edit | edit source]

Google WER Whisper WER
1272-128104-0000.flac 17.65 5.88
1272-128104-0001.flac 30.0 10.0
1272-128104-0002.flac 15.62 3.12
1272-128104-0003.flac 29.17 8.33
1272-128104-0004.flac 29.41 16.18
1272-128104-0005.flac 11.11 5.56
1272-128104-0006.flac 41.67 16.67
1272-128104-0007.flac 5.26 5.26
1272-128104-0008.flac 36.36 18.18
1272-128104-0009.flac 37.21 9.3
1272-128104-0010.flac 20.0 6.67
1272-128104-0011.flac 50.0 5.88
1272-128104-0012.flac 22.22 0.0
1272-128104-0013.flac 27.78 5.56
1272-128104-0014.flac 80.0 40.0

Detailed Transcriptions[edit | edit source]

Reference Google transcription Whisper transcription
1272-128104-0000.flac MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL Mr quilter is the apostle of the middle classes and we're glad to welcome his gospel Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
1272-128104-0001.flac NOR IS MISTER QUILTER'S MANNER LESS INTERESTING THAN HIS MATTER nor is Mr coaches Manor less interesting than his matter Nor is Mr. Quilter's manner less interesting than his matter.
1272-128104-0002.flac HE TELLS US THAT AT THIS FESTIVE SEASON OF THE YEAR WITH CHRISTMAS AND ROAST BEEF LOOMING BEFORE US SIMILES DRAWN FROM EATING AND ITS RESULTS OCCUR MOST READILY TO THE MIND he tells us that at this festive season of the Year with Christmas roast beef looming before us similies drawn from eating and it's results are coronavirus ready to the mind He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similarly drawn from eating and its results occur most readily to the mind.
1272-128104-0003.flac HE HAS GRAVE DOUBTS WHETHER SIR FREDERICK LEIGHTON'S WORK IS REALLY GREEK AFTER ALL AND CAN DISCOVER IN IT BUT LITTLE OF ROCKY ITHACA is grave doubts whether sir Frederic Layton's work is really Greek after all and can discover in it but little of Rocky is a car He has graved doubts whether Sir Frederick Layton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.
1272-128104-0004.flac LINNELL'S PICTURES ARE A SORT OF UP GUARDS AND AT EM PAINTINGS AND MASON'S EXQUISITE IDYLLS ARE AS NATIONAL AS A JINGO POEM MISTER BIRKET FOSTER'S LANDSCAPES SMILE AT ONE MUCH IN THE SAME WAY THAT MISTER CARKER USED TO FLASH HIS TEETH AND MISTER JOHN COLLIER GIVES HIS SITTER A CHEERFUL SLAP ON THE BACK BEFORE HE SAYS LIKE A SHAMPOOER IN A TURKISH BATH NEXT MAN Lozells pictures of gods and Adam paintings and Masons exquisite Angels national as a jingle poem Mr broken Fosters landscapes smile at one match in the same way that Mr Kaka used to flash his teeth and Mr John Collier gives his sister a cheerful slap on the back before he said like a shampooer in a Turkish bath next Man Lynelle's pictures are a sort of upgards and atom paintings, and Mason's exquisite ittles are as national as a jingo poem. Mr. Birkett Foster's landscapes smile at one much in the same way that Mr. Carker used to flash his teeth. And Mr. John Collier gives his sitter a cheerful slap on the back, before he says, like a shampooer and a Turkish bath. Next man!
1272-128104-0005.flac IT IS OBVIOUSLY UNNECESSARY FOR US TO POINT OUT HOW LUMINOUS THESE CRITICISMS ARE HOW DELICATE IN EXPRESSION is obviously unnecessary for us to point out how luminous these criticisms are how delicate an expression It is obviously unnecessary for us to point out how luminous these criticisms are, how delicate and expression.
1272-128104-0006.flac ON THE GENERAL PRINCIPLES OF ART MISTER QUILTER WRITES WITH EQUAL LUCIDITY and the general principles of artistic water rates with equal lucidity On the general principles of art, Mr. Krilter writes with equal lucidity.
1272-128104-0007.flac PAINTING HE TELLS US IS OF A DIFFERENT QUALITY TO MATHEMATICS AND FINISH IN ART IS ADDING MORE FACT painting he tells us is of a different quality to Mathematics and finish in art is adding more fat Painting, he tells us, is of a different quality to mathematics, and finish in art is adding more factor.
1272-128104-0008.flac AS FOR ETCHINGS THEY ARE OF TWO KINDS BRITISH AND FOREIGN as for itching there are two kinds of British and foreign As for etchings, there are two kinds, British and foreign.
1272-128104-0009.flac HE LAMENTS MOST BITTERLY THE DIVORCE THAT HAS BEEN MADE BETWEEN DECORATIVE ART AND WHAT WE USUALLY CALL PICTURES MAKES THE CUSTOMARY APPEAL TO THE LAST JUDGMENT AND REMINDS US THAT IN THE GREAT DAYS OF ART MICHAEL ANGELO WAS THE FURNISHING UPHOLSTERER elements most beautiful that has been made between decorative part and what we usually call pictures makes a customer appeal to The Last Judgement and reminds us that integrate days of art Michelangelo was the furnishing a poster He laments most bitterly the divorce that has been made between decorative art and what we usually call pictures. Mixed a customary appeal to the last judgment and reminds us that in the great days of art Michelangelo was the furnishing upholsterer.
1272-128104-0010.flac NEAR THE FIRE AND THE ORNAMENTS FRED BROUGHT HOME FROM INDIA ON THE MANTEL BOARD near the fire any ornaments Fred brought home from India on the mental board near the fire, and the ornaments Fred brought home from India on the mental board.
1272-128104-0011.flac IN FACT HE IS QUITE SEVERE ON MISTER RUSKIN FOR NOT RECOGNISING THAT A PICTURE SHOULD DENOTE THE FRAILTY OF MAN AND REMARKS WITH PLEASING COURTESY AND FELICITOUS GRACE THAT MANY PHASES OF FEELING in fact is quite severe Mr Ruskin from not recognising that I picked you shouldn't up the 3rd of man and remarks with please increase the infelicitous Grace that many faces is a feeling In fact, he is quite severe on Mr. Ruskin for not recognizing that a picture should denote the frailty of man, and remarks with pleasing courtesy and felicitous grace that many phases of feeling
1272-128104-0012.flac ONLY UNFORTUNATELY HIS OWN WORK NEVER DOES GET GOOD only unfortunately is only work never does get good Only, unfortunately, his own work never does get good.
1272-128104-0013.flac MISTER QUILTER HAS MISSED HIS CHANCE FOR HE HAS FAILED EVEN TO MAKE HIMSELF THE TUPPER OF PAINTING Mr Coulter has missed his chance for his failed even to make himself the tower of painting Mr. Quilter has missed his chance, for he has failed even to make himself the Tupper of Painting.
1272-128104-0014.flac BY HARRY QUILTER M A hi Harry quarter ma by Harry quilter MA.

Analysis[edit | edit source]

Abbreviations and different spellings cause many misjudgements in calculating WER. For example, in 0000, both Google and Whisper transcribed "Mister" as "Mr." and Google transcribed "we are" as "we're", which lead to 5.88 WER and 17.65 WER respectively. Same situation happened in 0001 and 0013 of Whisper transcriptions where "Mister" has been written as "Mr.". Different spellings, for example in Whisper 0011 where "recognising" has been written as "recognizing", together with "Mr.", leading to 5.88 WER. Since "Mr." and "Mister", "recognizing" and "recognising", "we are" and "we're" have no difference at all except for their spellings, we should consider above-mentioned transcriptions as correct with 0 WER.

Conclusion[edit | edit source]

The SR model from Whisper markedly outshines that of Google. Beyond just its superior WER, it delivers transcriptions that are more human-friendly, complete with fitting punctuation. Consequently, Whisper's results integrate better with NLP models, such as machine translation.

To enhance the precision of the WER evaluation, we could better normalize both the references and hypotheses before evaluating WER. For instance, equating terms like 'mister' and 'mr' would aid in consistency.

Comparison Between Transcriptions and Original Transcript using whisper[edit | edit source]

Xueying LIU[edit | edit source]

https://drive.google.com/file/d/1jrPuAcyzeAFT4cg0Lc1nJg115sqP5yCu/view?usp=drive_link

Whisper WER Whisper transcription
1272-128104-0000.flac 100 감사합니다.
1272-128104-0001.flac 10.0 Nor is Mr. Culture's manner less interesting than his matter.
1272-128104-0005.flac 5.56 It is obviously unnecessary for us to point out how luminous these criticisms are, how delicate and expression.
1272-128104-0006.flac 16.67 On the general principles of art, Mr. Kult writes with equal lucidity.
1272-128104-0007.flac 5.26 Painting, he tells us, is of a different quality to mathematics and finish in art is a more fact.
1272-128104-0010.flac 13.33 Near the fire and the ominous, Fred brought home from India on the mental board.
1272-128104-0012.flac 0.0 Only unfortunately his own work never does get good.
1272-128104-0013.flac 11.11 Mr. Quilter has missed his chance for he has failed even to make himself the tougher of painting.

Recording Device: iPhone13

Recording Environment: Noisy, normal speed

It is surprising to see that 0000 was transcribed into Korean. Compared with the WER of original audio recognized by Whisper, 5 out of 8 sentences from this speaker had the same WER with original audio, and 3 had higher WER than original audio. Reasons for the low performance could be that first, these audios were recorded under noisy environment with people talking in the background; second, some words in the sentences are uncommon and might be mispronounced by the speaker.

Jingxuan YUE[edit | edit source]

https://drive.google.com/file/d/1UYOKGkM0cli9htQ6ry6SLuE2Wa2_b3-U/view?usp=drive_link

Whisper WER Whisper transcription
1272-128104-0000.flac 35.29 Mr. Krueger is a parcel of the middle classes and we're glad to welcome his gospel.
1272-128104-0001.flac 30 Nor is Mr. Quotor's manner less interesting than his manner.
1272-128104-0002.flac 18.75 He tells us that at this festive season of the year, with Christmas and Rosebeef looming before us, similarly drowned from 18 and its resounds occur most readily to the mind.
1272-128104-0003.flac 37.5 He has graved out the weather, so Frederick Lydon's work is really great after all and can discover the internet but little of Rocky Ithaca.
1272-128104-0004.flac 30.88 The Niles pictures are a sort of op-cards and atom paintings and maisons. Equasites, idols are as national as a jingle poem. Mr. Borke Foster's landscapes smile at one mudge in the same way that Mr. Cocker used to flash his teeth and Mr. Drong. Cocker gives his sitter a cheerful slap on the back before he says like shampoo in a Turkish bath next to man.
1272-128104-0005.flac 11.11 It is obviously a necessary for us to point out how luminous these criticisms are, how delicate in expression.
1272-128104-0006.flac 16.67 On the general principles of art, Mr. Quilter writes word equal lucidity.
1272-128104-0007.flac 10.53 Painting he tells us is of a different quality to mathematics and finishing art is adding more fact.
1272-128104-0008.flac 9.09 As for addings, they are of two kinds, British and foreign.
1272-128104-0009.flac 0 He laments most bitterly the divorce that has been made between decorative art and what we usually call pictures makes the customary appeal to the last judgment and reminds us that in the great days of art. Michael Angelo was the furnishing upholsterer.
1272-128104-0010.flac 20 Near the fire and ornaments fled brought home from India on the mental board.
1272-128104-0011.flac 17.65 In fact, he is quite severe on Mr. Ruskin for not recognizing that a picture should denote the frailty of men and remarks with pleasing courtesy and the felicitous grace that many faces a feeling.
1272-128104-0012.flac 0 Only, unfortunately, his own work never does get good.
1272-128104-0013.flac 11.11 Mr. Quilter has missed his chance for he has failed even to make himself the tougher of painting.
1272-128104-0014.flac 40 by Harry Quilter, M.A.

Recording device: iPhone XR

Environment: relatively quiet corridor with occasional background noise, standard speech rate

Notes:

In general, the WER in this transcription is considerably higher than that of the original audiobook transcribed by Whisper. The primary reason behind this discrepancy lies in the fact that this recording was made by a native Mandarin speaker, and there are instances where the pronunciation of certain phonetics is not entirely accurate when speaking in continuous sentences. For example, words like 'man' and 'men,' which have similar pronunciations, lead to a higher WER.

However, there are a few exceptions to this trend. Notably, the WER for segments 0008 and 0009 in this transcription is lower than that of the original recording. This is especially evident in segment 0009, which features a lengthy sentence with no transcription errors. This indicates that Whisper's Large Vocabulary Continuous Speech Recognition (LVCSR) system performs admirably in specific cases.

There are instances where Whisper faces challenges. Firstly, it struggles when dealing with words that share the same or similar pronunciations. For instance, in segment 0003, the original word 'WHETHER' is transcribed as 'weather.' Another instance occurs in segment 0011, where 'PHASES' is transcribed as 'faces.' This highlights the need for improvement in the system's ability to transcribe within the context of the language model.

Additionally, the transcription quality suffers when it comes to liaison. For example, in segment 0000, the original transcript reads 'apostle,' but the transcription results in 'a parcel.' A similar situation arises in segment 0004, where 'AT EM' is transcribed as 'atom.'

M.Tepei[edit | edit source]

https://drive.google.com/file/d/1ysPduZXsf_pUziEq_-p1cqUv8EGULc03/view?usp=drive_link

Whisper WER Whisper transcription
1272-128104-0000.flac 41.18 The answer is the question of the middle classes and we are going to let the well-composed gospel.
1272-128104-0001.flac 20 Nor is Mr. Quilter's manner less interesting than this matter.
1272-128104-0002.flac 18.75 The process that at this festive season of the year with Christmas and roast beef looming before us, similes drawn from eating and its result occur more spreadly to the mind.
1272-128104-0003.flac 20.83 He has graved out sweater Sir Frederick Layton's work is really Greek after all and can discover in it but little Rocky Ithaca.
1272-128104-0004.flac 17.65 The pictures are a sort of upgarts and at paintings and mason's exquisite idls are as national as a jingle poem Mr. Burkid Foster's landscape smile. At one much in the same way that Mr. Carker used to flash his teeth and Mr. John Collier gives his sitter a cheerful slap on the bag before he says like a shampooer in a Turkish bath next man.
1272-128104-0005.flac 5.56 It is obviously unnecessary for us to point out how luminous these criticisms are, how delicate and expression.
1272-128104-0006.flac 33.33 The general principles of art, Mr. Quilder, rise with equal lucidity.
1272-128104-0007.flac 15.79 I think he tells us as of a different quality to mathematics and finish in art is adding more fact.
1272-128104-0008.flac 0 As for etchings, they are of two kinds, British and foreign.
1272-128104-0009.flac 6.98 He laments most bitterly the divorce that has been made between decorative art and what we usually call pictures makes the customary appeal to the last judgment and reminds us that in the great days of art Michelangelo was the furnishing upholster.
1272-128104-0010.flac 40 The entire empire and ornaments thread brought home from India on the mental board.
1272-128104-0011.flac 11.76 In fact, he is quite severe on Mr. Ruskin for not recognizing that a picture should denote the frailty of men and remarks with pleasing courtesy and felicitous grace that many faces of feeling.
1272-128104-0012.flac 0 Only unfortunately his own work never does get good.
1272-128104-0013.flac 16.67 Mr. Guilter has missed his chance for he has failed even to make himself the upper of painting.
1272-128104-0014.flac 60 by every quilter ma.

Recording device: Dell XPS 13

Environment: quiet, modal voice, standard (natural) speech rate.

Notes: Interesting to see that 'Mr. Quilter' is transcribed correctly once, and with different minor errors for the rest of the cases. Did not expect 'etchings' to be transcribed correctly, as I hesitated/stuttered when I pronounced it.

Contributors[edit | edit source]

  • Yuxing OUYANG
  • Xiaoling LIN
  • Xueying LIU
  • Jingxuan YUE
  • M.Tepei