Version 1
: Received: 28 June 2024 / Approved: 28 June 2024 / Online: 2 July 2024 (03:02:10 CEST)
How to cite:
Tits, N.; Bhatnagar, P.; Dutoit, T. TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer. Preprints2024, 2024062082. https://doi.org/10.20944/preprints202406.2082.v1
Tits, N.; Bhatnagar, P.; Dutoit, T. TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer. Preprints 2024, 2024062082. https://doi.org/10.20944/preprints202406.2082.v1
Tits, N.; Bhatnagar, P.; Dutoit, T. TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer. Preprints2024, 2024062082. https://doi.org/10.20944/preprints202406.2082.v1
APA Style
Tits, N., Bhatnagar, P., & Dutoit, T. (2024). TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer. Preprints. https://doi.org/10.20944/preprints202406.2082.v1
Chicago/Turabian Style
Tits, N., Prernna Bhatnagar and Thierry Dutoit. 2024 "TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer" Preprints. https://doi.org/10.20944/preprints202406.2082.v1
Abstract
In this paper, we present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (wav2vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained thanks to forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English, respectively. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems. We leave experiments on other languages for future work but the design of the system makes it easily adaptable to other languages.
Keywords
speech recognition; phoneme recognition; deep learning; transfer learning
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
