Abstract |
L1 Shadowing L2 Concept |
Evaluating speech intelligibility is a critical task in computer-aided language learning systems. Traditional methods often rely on word error rates (WER) provided by automatic speech recognition (ASR) as intelligibility scores. However, this approach has significant limitations due to notable differences between human speech recognition (HSR) and ASR. A promising alternative is to involve a native (L1) speaker in shadowing what nonnative (L2) speakers say. Breakdowns or mispronunciations in the L1 speaker’s shadowing utterance can serve as indicators for assessing L2 speech intelligibility. In this study, we propose a speech generation system that simulates the L1 shadowing process using voice conversion (VC) techniques and latent speech representations. Our experimental results demonstrate that this method effectively replicates the L1 shadowing process, offering an innovative tool to evaluate L2 speech intelligibility. Notably, systems that utilize self-supervised speech representations (S3R) show a higher degree of similarity to real L1 shadowing utterances in both linguistic accuracy and naturalness. |
|
|---|
Native Speaker Shadowing System with Latent Speech Representations |
2 Step Training Methods |
|
|
|---|
Audio Samples (Click ID to display reference text)
THE CHILDREN WHO ARE THE SAME GENERATIONS ARE SUFFERING FROM HUNGER.
| Source | Target | Reference | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(L2_{R}\) | Mel. | PPG-BNF | \(L1_{S1}\) | \(L1_{SS}\) | |||||||
| \(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.FT |
\(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.Ft |
||||
| PPG-BNF | HuBERT | ||||||||||
ON THE OTHER HAND I'M LOOKING FORWARD TO EXPERIENCE THEIR DAILY LIFE BECAUSE I ASPIRE TO LIVING OVERSEAS
| Source | Target | Reference | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(L2_{R}\) | Mel. | PPG-BNF | \(L1_{S1}\) | \(L1_{SS}\) | |||||||
| \(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.FT |
\(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.Ft |
||||
| PPG-BNF | HuBERT | ||||||||||
IT LEADS TO CONFIDENCE.
| Source | Target | Reference | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(L2_{R}\) | Mel. | PPG-BNF | \(L1_{S1}\) | \(L1_{SS}\) | |||||||
| \(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.FT |
\(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.Ft |
||||
| PPG-BNF | HuBERT | ||||||||||
IT IS IMPORTANT TO IDENTIFY THE CORRECT SOURCE.
| Source | Target | Reference | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(L2_{R}\) | Mel. | PPG-BNF | \(L1_{S1}\) | \(L1_{SS}\) | |||||||
| \(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.FT |
\(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.Ft |
||||
| PPG-BNF | HuBERT | ||||||||||
IT IS FULL OF FEELING THAT YOU ARE WALKING IN THE CITY.
| Source | Target | Reference | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(L2_{R}\) | Mel. | PPG-BNF | \(L1_{S1}\) | \(L1_{SS}\) | |||||||
| \(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.FT |
\(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.Ft |
||||
| PPG-BNF | HuBERT | ||||||||||
IN PETER'S CHILDHOOD HE WAS ORIGINALLY ABANDONED AND NOT HEALTHY BUT NOW HE IS A CHEERFUL BOY AND A GOOD BROTHER WHO LOOKS AFTER THE YOUNGER BROTHER WELL NEXT THIS IS BIEBER
| Source | Target | Reference | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(L2_{R}\) | Mel. | PPG-BNF | \(L1_{S1}\) | \(L1_{SS}\) | |||||||
| \(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.FT |
\(L1_{S1}\) | \(L1_{SS}\) | \(L1_{S1}\) Enc.FT |
\(L1_{S1}\) Dec.Ft |
||||
| PPG-BNF | HuBERT | ||||||||||