Version 1
: Received: 29 September 2024 / Approved: 30 September 2024 / Online: 30 September 2024 (14:37:06 CEST)
How to cite:
Peng, J.; Zhong, K. Accelerating and Compressing Transformer-based PLMs for Enhanced Comprehension of Computer Terminology. Preprints2024, 2024092415. https://doi.org/10.20944/preprints202409.2415.v1
Peng, J.; Zhong, K. Accelerating and Compressing Transformer-based PLMs for Enhanced Comprehension of Computer Terminology. Preprints 2024, 2024092415. https://doi.org/10.20944/preprints202409.2415.v1
Peng, J.; Zhong, K. Accelerating and Compressing Transformer-based PLMs for Enhanced Comprehension of Computer Terminology. Preprints2024, 2024092415. https://doi.org/10.20944/preprints202409.2415.v1
APA Style
Peng, J., & Zhong, K. (2024). Accelerating and Compressing Transformer-based PLMs for Enhanced Comprehension of Computer Terminology. Preprints. https://doi.org/10.20944/preprints202409.2415.v1
Chicago/Turabian Style
Peng, J. and Kai Zhong. 2024 "Accelerating and Compressing Transformer-based PLMs for Enhanced Comprehension of Computer Terminology" Preprints. https://doi.org/10.20944/preprints202409.2415.v1
Abstract
Pre-trained language models (PLMs) have significantly advanced natural language processing (NLP), establishing the "pre-training + fine-tuning" paradigm as a cornerstone approach in the field. However, the vast size and computational demands of Transformer-based PLMs present challenges, particularly regarding storage efficiency and processing speed. This paper addresses these limitations by proposing a novel lightweight PLM optimized for accurately understanding domain-specific computer terminology. Our method involves a pipeline parallelism algorithm designed to accelerate training. It is paired with an innovative mixed compression strategy that combines pruning and knowledge distillation to effectively reduce the model size while preserving its performance. The model is further fine-tuned using a dataset that mixes source and target languages to enhance its versatility. Comprehensive experimental evaluations demonstrate that the proposed approach successfully achieves a balance between model efficiency and performance, offering a scalable solution for NLP tasks involving specialized terminology.
Computer Science and Mathematics, Computer Science
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
