Preprint
Article
Towards Bengali Word Embedding: Corpus Creation, Intrinsic and Extrinsic Evaluations
Altmetrics
Downloads
445
Views
492
Comments
0
This version is not peer-reviewed
Submitted:
22 December 2020
Posted:
23 December 2020
You are already at the latest version
Alerts
Abstract
Distributional word vector representation orword embedding has become an essential ingredient in many natural language processing (NLP) tasks such as machine translation, document classification, information retrieval andquestion answering. Investigation of embedding model helps to reduce the feature space and improves textual semantic as well as syntactic relations.This paper presents three embedding techniques (such as Word2Vec, GloVe, and FastText) with different hyperparameters implemented on a Bengali corpusconsists of180 million words. The performance of the embedding techniques is evaluated with extrinsic and intrinsic ways. Extrinsic performance evaluated by text classification, which achieved a maximum of 96.48% accuracy. Intrinsic performance evaluatedby word similarity (e.g., semantic, syntactic and relatedness) and analogy tasks. The maximum Pearson (ˆr) correlation accuracy of 60.66% (Ssˆr) achieved for semantic similarities and 71.64% (Syˆr) for syntactic similarities whereas the relatedness obtained 79.80% (Rsˆr). The semantic word analogy tasks achieved 44.00% of accuracy while syntactic word analogy tasks obtained 36.00%
Keywords:
Subject: Engineering - Automotive Engineering
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
MDPI Initiatives
Important Links
© 2024 MDPI (Basel, Switzerland) unless otherwise stated