Professional Documents
Culture Documents
Thibault Roux InterSpeech 2022 v2
Thibault Roux InterSpeech 2022 v2
Formally, the WER is calculated as follows: Developed for text generation [7], this metric aims at compar-
ing a reference word and a hypothesis with respect to semantic
#S + #I + #D proximity. The first step consists in obtaining the words and
W ER = (1) sub-words (tokens) of the reference and the hypothesis thanks
#ref erence words
By definition, the WER therefore considers any type of er- to the WordPiece tokenizer used by BERT [8].
ror of equivalent importance. This is the main advantage of this Then, given the sequence of contextualized embeddings of
metric: its simplicity of application and use. However, the WER reference (x1,...,xk) and hypothesis (x̂1,...,x̂m), the cosine sim-
does have limitations. Using the previous example, the word ilarity is computed between each reference and hypothesis em-
Patrick was transcribed as playing. An alternative transcription beddings to obtain a score matrix weighted here with the inverse
hypothesis could have been Patricia. In both cases, the WER frequency of the document [7].
would be identical to the reference, even though the nature of To compute the precision, we associate each token x with
the error is different (Patricia is in the same grammatical cate- a token x̂ by selecting the token bringing the highest similar-
gory while playing is different from the reference word in terms ity. The recall is computed by associating each x̂ token with
of syntactics and semantics). Another limitation concerns the an x token in the same way. The f-measurement score, which
few categories considered (substitution, insertion, deletion) for we use in our experiments, is computed with the recall and the
the rate calculation carrying no additional information about the precision [7].
context.
2.2. Character Error Rate (CER) 2.7. Sentence Semantic Distance (SemDist)
The character error rate (CER) is based on the same principle as While previous metrics focus on words and characters, the prin-
the WER but applied to character chains instead of word chains. ciple of this metric [9] is to consider the complete sentence. In
It has already been used in the ASR domain [6]. Initially, it is the ASR framework, the reference and the hypothesis are re-
particularly suitable for character-based languages such as Chi- spectively transformed into their sentence embeddings using a
nese or Japanese. For Latin languages, and in particular French, SentenceBERT [10] model, i.e. a model of sentence embed-
the CER allows, among other things, to give an indication of the dings using the contextual word embeddings of BERT [8]. It is
nature of the errors: a low CER could indicate that the ASR sys- then possible to compare these vectors with the cosine similar-
tem tends to generate words close to the reference (and thus po- ity. Our final measure is the average of the cosine similarities
tentially incorporating errors related to gender, number, tense, between each reference’s sentence embeddings and its respec-
etc.) as opposed to a high CER, with transcription assumptions tive hypothesis.
that are very distant from the references.
3.2. Automatic Speech Recognition (ASR) system we use the default multilingual-BERT base model4 .
The ASR system is based on an existing state-of-the-art recipe1
that uses the Kaldi [16] toolkit. The acoustic model is a deep 4. Experiments and Analysis
neural network based on the TDNNF [17] architecture. To make
the system more robust to different acoustic conditions, the au- This section presents firstly an analysis of the six applied met-
dio files were randomly perturbed in speed and volume (i.e. data rics presented in Section 2 in addition to the WER, and secondly
augmentation) during the training process. a qualitative study of the impact of the language model rescor-
Three language models are used. The first is a trigram ing process used in our ASR system.
model trained with SRILM [18] and used directly by the ASR
system. The second is a RNNLM, a deep neural network based 4.1. Metrics analysis
language model, used in an a posterior rescoring process. The
network consists of three TDNN layers interspersed with two In order to make a more in-depth analysis of our metrics, in par-
LSTM layers. Also, a quadrigram model is used during the ticular to understand and estimate the links that they can main-
rescoring step. The training corpus and the vocabulary used to tain between them, we calculated a Pearson correlation between
learn the trigram model, the RNNLM model and the quadri- our different measurements for our two systems and averaged
gram model are identical. The rescoring is optional as we want them in Table 1. The higher the score between two metrics, the
to observe its impact on the different metrics. more they are considered correlated. Clearly, the first remark is
that not all metrics correlate with each other in the same way.
3.3. Tools SemDist is the metric that correlates the least with the others.
This might be due to the fact that it is the only metric based
We used the POET tool2 , a POS tagger for French language on sentence embeddings in our experiments, going beyond the
based on Flair [19] contextual embeddings and used to automat- word dimension. This weak correlation implies that minimiz-
ically extract the morpho-syntactic information from words. We ing the WER would not correlate strongly with better perfor-
chose this labeler because it allows us to have both the generic mance on downstream tasks (i.e. extrinsic evaluation) using
classes of Universal Dependency (noun, adjective, adverb, etc.) sentence embeddings. This idea is consistent with many pub-
but also a fine granularity thanks to additional information on lications in NLP and ASR that consider intrinsic ratings to be
these same labels (feminine plural noun, third person plural less relevant than extrinsic ratings [21, 22]. Indeed, the authors
personal pronoun, etc.). We then propose two measures based of SemDist [9] concluded that their metric correlated better with
on POS tags derived from the POSER (Section 2.3): one inte- downstream tasks than the WER.
grating the detailed classes (dPOSER) and one with the generic
classes of Universal Dependency (uPOSER). Note that no man- We can see that the metric that correlates best with
ual POS tag annotation was used: both reference and hypothesis BERTScore and SemDist is EmbER, all three of which are
transcripts were automatically tagged. To obtain the lemmas, based on embeddings, while the metric that correlates best with
we used the Spacy lemmatizer for French3 . EmbER is WER. This highlights that the Embedding Error Rate
is a hybrid metric that has the advantage of correlating with
For the EmbER metric (Section 2.5), we used Fasttext em-
WER and embeddings-based metrics.
beddings [20] and applied an error of 0.1 if the cosine simi-
larity is above a threshold of 0.4, and 1 in other cases. The An interesting observation to make is that LER correlate the
threshold was decided empirically given the cosine similarity best with uPOSER and has a better correlation with dPOSER
between synonyms compare to cosine similarity between words than LCER. It seems that part-of-speech and lemmas share
randomly chosen. some similarity : if the lemma is wrong, the POS is often wrong.
For the SemDist metric (Section 2.7), the multilingual Sen- Also, the LCER and the CER have a correlation of 0.9831 which
tenceBERT embeddings was used. Finally, for the BERTScore, probably means that when the CER is high, there is a good
chance that the word is wrong too and so is the lemma. On the
1 https://github.com/kaldi-asr/kaldi/blob/
other hand, it also means that the LCER does not bring more
information than the CER.
master/egs/librispeech/s5/
2 https://huggingface.co/qanastek/pos-french
3 https://github.com/explosion/spacy-models/
releases/tag/fr_core_news_md-3.2.0 4 https://github.com/Tiiiger/bert_score
System WER CER dPOSER uPOSER LER LCER SemDist BERTScore EmbER
Base 15.45 8.57 14.59 12.22 14.35 8.78 7.89 9.12 12.33
Rescoring 13.24 7.70 12.51 10.79 12.08 8.00 7.18 8.38 10.79
Reduction -14.3 % -10.2 % -14.3 % -11.7 % -15.8 % -8.8 % -9.0 % -8.1 % -12.5 %
Table 2: Performance comparison of the Base and Rescoring systems using different metrics. The observed reduction between the two
systems, in relative value, is also provided.