1. Introduction

2. Literature Review

2.2. Deep Learning Based Approaches

This research paper [15] addressed the exponential growth of social media, which, while empowering free expression, also facilitated online harassment and hate speech. It highlighted the lack of computational resources for under-resourced languages like Bengali. In response, it developed BengFastText, the largest Bengali word embedding model, based on 250 million articles. It created three extensive datasets for hate speech detection, document classification, and sentiment analysis. Experiments with a Multichannel Convolutional LSTM (MC-LSTM) network, incorporating BengFastText, demonstrated superior performance compared to baseline models, achieving high F1-scores. Similarly, another research paper [4] introduced the impact of social media platforms and online streaming services on the proliferation of hate speech. To address the need for linguistically diverse datasets, it introduced BD-SHS, a large manually labeled dataset including hate speech in various social contexts. BD-SHS contains over 50,200 offensive comments, making it significantly larger than previous Bangla hate speech datasets. The dataset was annotated using a hierarchical process that included hate speech identification, target identification, and categorization of hate speech types. For benchmarking, various models, including SVM and Bi-LSTM architectures, were experimented with. The Bi-LSTM model, trained with informal embeddings derived from 1.47 million social media comments, achieved the highest F1-score of 91.0% in hate speech identification. This model consistently outperformed other pre-trained embeddings like BengFastText and multilingual fastText, which were trained on formal texts. Another study [1] highlighted the growing prevalence of hate speech on social media, particularly in Bengali, despite extensive research in other languages. It addressed this gap by proposing an encoder-decoder-based machine learning model to classify Bengali comments on Facebook. It collected a dataset of 7,425 comments across seven hate speech categories, using a combination of automated and manual methods due to limitations with the Facebook Graph API. The preprocessing involved tokenization, stemming, stopword removal, and extracting features using TF-IDF and word embedding. Three models—LSTM, GRU, and attention-based decoders—were evaluated, with the attention-based model achieving the highest accuracy at 77%. The study also incorporated a Bangla Emot Module to detect emotions from emojis and emoticons, enhancing the model’s interpretability. Furthermore, another research paper [16] investigated the detection and classification of hateful speech in Bengali language social media, focusing on Facebook comments. The study employed both traditional machine learning algorithms and a GRU-based deep neural network model. It compiled and annotated a dataset of 5,126 comments, categorizing them into six classes: Hate Speech, Communal Attack, Inciteful, Religious Hatred, Political Comments, and Religious Comments. This dataset marks the first significant contribution to Bengali language hate speech detection in social media. Comparing various machine learning algorithms, it achieved 52.20% accuracy with Random Forest, which improved to 70.10% using the GRU model. The study highlighted the importance of linguistic and quantitative feature extraction tailored to the Bengali social context and underscored the superior performance of the GRU model in understanding context and semantics, which is critical for accurate hate speech detection in Bengali. Collectively, these studies emphasize the need for more advanced computational models and resources tailored to the Bengali language, highlighting the absence of LLMs in addressing hate speech detection comprehensively.

Table 1. Summary of Studies on Hate Speech Detection in Bengali Language

Table 1. Summary of Studies on Hate Speech Detection in Bengali Language

Types	Authors	Year	Models Employed	Performance Metrics	Key Findings
Traditional Approaches	Manash et al. [12]	2022	Gated Recurrent Unit (GRU), Logistic Regression, Random Forest, Multinomial Naive Bayes (MNB), Support Vector Machine (SVM)	GRU: 78.89% accuracy, MNB: 80.51% accuracy	Developed a dataset of 2,000 Bengali comments; highlighted scarcity of Bengali datasets and importance of context-specific feature extraction; MNB and GRU models effective in detecting anti-social comments.
	Sherin et al. [13]	2022	Logistic Regression, Multinomial Naive Bayes, Random Forest, Support Vector Machine (SVM), Gradient Boosting	SVM: 85.7% accuracy	Dataset of 5,000 comments; emphasized challenges in multi-class classification for Bangla; binary classification was used; highlighted importance of data preprocessing and TFIDF feature extraction.
	Istiaq et al. [14]	2021	Logistic Regression, Gated Recurrent Unit (GRU)	GRU: 98.89% accuracy	Created dataset from scratch with videos from YouTube; high accuracy with GRU model; logistic regression also showed high precision, recall, and F1 scores; focused on detecting hate speech in Bangla videos.
Deep Learning Approaches	Rezaul et al. [15]	2020	Multichannel Convolutional LSTM (MConv-LSTM), incorporating BengFastText	MConv-LSTM: F1-scores of 90.45%	Developed BengFastText, the largest Bengali word embedding model based on 250 million articles; created three extensive datasets; MC-LSTM with BengFastText outperformed baseline models.
	Nauros et al. [4]	2022	Bi-LSTM, Support Vector Machine (SVM)	Bi-LSTM: F1-score of 91.0%	Introduced BD-SHS, a large manually labeled dataset with over 50,200 offensive comments; Bi-LSTM trained with informal embeddings achieved highest F1-score; outperformed other pre-trained embeddings like BengFastText and MFT.
	Amit et al. [1]	2022	LSTM, GRU, Attention-based decoders	Attention-based model: 77% accuracy	Proposed an encoder-decoder-based model for classifying Bengali Facebook comments; collected 7,425 comments across seven hate speech categories; attention-based model achieved highest accuracy; included Bangla Emot Module
	Alvi et al. [16]	2019	GRU, Random Forest	GRU: 70.10% accuracy	Compiled and annotated a dataset of 5,126 comments into six classes; Random Forest achieved 52.20% accuracy, GRU model improved to 70.10%; emphasized importance of linguistic and quantitative feature extraction for Bengali.

2.3. Transformer Based Approaches

This research paper [3] emphasizes the critical need to address the proliferation of hate speech on social media in Bangladesh, particularly due to the lack of comprehensive Bangla datasets. It compiled a new dataset of 8600 user comments from Facebook and YouTube, categorized into sports, religion, politics, entertainment, and others. Various models were tested for hate speech detection, with BERT showing the highest accuracy at 80% on their dataset. When applied to an existing dataset of 30,000 records, BERT achieved an impressive accuracy of 97%, surpassing the performance of previously tested models like SVM, LSTM, and BiLSTM. However, there is a noticeable absence of Large Language Models specifically tailored for Bangla hate speech detection. Furthermore, another research paper [5] highlights that social media is a hotspot for hateful and offensive content, impacting race, gender, and religion in an unprejudiced society. Despite significant research on hate speech detection in English, there’s a notable gap in low-resource languages like Bengali, including its Romanized form used in social media interactions. To address this, it developed an annotated dataset of 10K Bengali posts (5K actual and 5K Romanized) and implemented several baseline models for classification. Experiments using m-BERT, XLM-Roberta, and IndicBERT were conducted, exploring interlingual transfer mechanisms. XLM-Roberta performed best when training datasets separately, while MuRIL outperformed in joint and few-shot training scenarios by better interpreting semantic expressions. Nonetheless, the absence of advanced Large Language Models for Bangla hate speech detection remains a significant gap. Moreover, this research paper [17] highlights the rapid expansion of social media and micro-blogging platforms, which have empowered freedom of expression but also facilitated the spread of antisocial behaviors such as online harassment, cyberbullying, and hate speech. The study evaluates four variants of the BERT model on a test set, reporting XML-RoBERTa as the top-performing model with an F1-score of 87%, outperforming other transformer models by 2% to 5%. Using WeightWatcher for ensemble prediction, they achieved the highest MCC score of 0.82, indicating a strong correlation between predictions and ground truths. Their ensemble approach improved overall accuracy by 1.8% across classes, effectively addressing misclassification rates. The effectiveness of BERT variants over traditional ML and DNN baselines was emphasized, particularly in minimizing classification errors across imbalanced datasets. The study provided class-specific classification reports, highlighting nuances in detecting personal versus political hate speech, with specific challenges noted in identifying political hate due to overlapping terms with personal hate expressions. It also explored feature selection’s impact on ML baselines, noting significant improvements in models like GBT, which outperformed others with an MCC score of 0.571. However, SVM, LR, and NB models showed degraded performance due to feature selection’s impact on their assumptions of feature independence. In contrast, CNN and Bi-LSTM among DNN baselines performed reasonably well, although they fell short compared to transformer-based models like XML-RoBERTa. Despite these advances, the lack of Large Language Models specifically designed for Bangla hate speech detection persists as a major limitation.

Table 2. Summary of Studies on Hate Speech Detection in Bengali Language

Table 2. Summary of Studies on Hate Speech Detection in Bengali Language

Types	Authors	Year	Models Employed	Performance Metrics	Key Findings
Transformer Based Approaches	Jobair et al. [3]	2023	BERT, SVM, LSTM, BiLSTM	BERT: 80% accuracy on new dataset, 97% accuracy on existing dataset	Compiled a dataset of 8600 comments; BERT showed highest accuracy at 80% on new dataset and 97% on existing dataset of 30,000 records; BERT outperformed SVM, LSTM, and BiLSTM.
	Mithun et al. [5]	2022	m-BERT, XLM-RoBERTa, IndicBERT, MuRIL	m-BERT: F1-score of 0.81	Developed an annotated dataset of 10K Bengali posts (5K actual, 5K Romanized); XLM-RoBERTa performed best in separate training; MuRIL outperformed in joint and few-shot training scenarios.
	Rezaul et al. [17]	2020	BERT variants (including XLM-RoBERTa), traditional ML models, DNN models (CNN, Bi-LSTM)	XLM-RoBERTa: F1-score of 87%, MCC score of 0.82	Evaluated BERT variants; XLM-RoBERTa achieved highest F1-score of 87%; ensemble approach improved overall accuracy by 1.8%; highlighted challenges in detecting political hate speech; traditional ML models showed varied performance due to feature selection.
Large Language Models	Keyan et al. [7]	2024	GPT-3.5-turbo, Chain-of-Thought prompts	Accuracy: 0.85, Precision: 0.8, Recall: 0.95, F1 Score: 0.87	Chain-of-Thought reasoning prompts significantly outperform other strategies, capturing intricate contextual details for accurate hate speech detection.
	Sarthak et al. [8]	2023	Flan-T5-large, text-davinci-003, GPT-3.5-turbo-0301	F1 Scores: Flan-T5-large: 0.59 (HateXplain), 0.63 (implicit hate), text-davinci-003: 0.45 (HateXplain), 0.36 (implicit hate)	Flan-T5-large outperforms other models with vanilla prompts. Incorporating target community information into prompts yields a 20-30% performance boost. Precise prompt engineering is critical for optimizing LLMs in hate speech detection.
	Flor et al. [9]	2023	mT0, FLAN-T5, multilingual XLM-RoBERTa	Macro-F1 Scores: FLAN-T5: 65.34 (English), 62.61 (Spanish), 57.29 (Italian)	Zero-shot learning with prompting can match or surpass fine-tuned models’ performance, particularly with instruction fine-tuned models. Prompt and model selection significantly impact accuracy.

3. Background Study

3.1. Transformer Based Models

3.1.1. BERT-based Transformer Models

Bidirectional Encoder Representations from Transformers, commonly known as BERT [18], is built upon a sophisticated deep learning framework that establishes connections between input and output elements, adaptively determining their relationships. BERT’s distinctive feature is its bidirectional training capability, which allows the model to understand the context of a word by considering both its preceding and succeeding words. This bidirectional approach contrasts with traditional models that only consider one direction, either forward or backward. BERT’s architecture is designed to enable the model to capture intricate linguistic patterns and contextual dependencies in text, making it exceptionally powerful for a variety of natural language processing tasks. BanglaBERT Base [19], a variant tailored for the Bengali language, follows the same robust architecture as the original BERT model. This alignment ensures that BanglaBERT Base inherits the powerful contextual understanding capabilities of BERT, enabling it to effectively process and analyze Bengali text with high accuracy. mBERT [20], or Multilingual BERT, extends the BERT architecture to support multiple languages. It is pre-trained on a large corpus of text from 104 different languages, including Bengali, enabling it to understand and generate text across various languages. mBERT leverages the same bidirectional training approach as BERT, allowing it to capture the context of words from diverse linguistic backgrounds. This makes mBERT a versatile tool for multilingual natural language processing tasks. XLM-RoBERTa [21], or Cross-lingual Language Model - RoBERTa, is an extension of the RoBERTa model designed to handle multiple languages. It is pre-trained on a vast dataset that includes text from 100 languages, which helps it learn cross-lingual representations. XLM-RoBERTa uses the same bidirectional training approach as BERT, allowing it to capture the context of words in a variety of languages. This makes it a powerful tool for multilingual tasks, such as cross-lingual understanding and translation.

3.1.2. ELECTRA-based Transformer Models

Efficiently Learning an Encoder that Classifies Token Replacements Accurately, known as ELECTRA [22], utilizes a unique pre-training task focused on identifying replaced tokens within an input sequence. This method involves two main components: a discriminator model and a generator model. The discriminator model is trained to recognize which tokens have been replaced in a corrupted sequence, while the generator model is simultaneously trained to predict the original tokens for the masked out ones. This setup is somewhat reminiscent of a generative adversarial network (GAN) training system, but without the adversarial component. In ELECTRA, the generator is not trained to deceive the discriminator, but rather to provide accurate token predictions. This collaborative process enhances the model’s ability to understand and generate text. BanglaBERT [23] serves as the ELECTRA discriminator model, specifically designed for the Bengali language. This model effectively leverages the ELECTRA architecture to accurately classify and process Bengali text, improving the overall performance of text classification tasks in the Bengali language.

3.1.3. ALBERT-based Transformer Models

A Lite BERT, known as ALBERT [24], has shown that exceptional language models do not always require larger architectures. ALBERT achieves efficiency and high performance by utilizing the same encoder segment architecture as the original Transformer but introduces three crucial modifications: factorized embedding parameters, cross-layer parameter sharing, and employing Sentence-order Prediction (SOP) instead of Next Sentence Prediction (NSP). These modifications allow ALBERT to maintain a smaller model size while still delivering superior performance. Factorized embedding parameters reduce the number of parameters, cross-layer parameter sharing ensures consistency and reduces redundancy across layers, and SOP provides a more challenging pre-training objective than NSP, leading to better model understanding and contextual awareness. In the context of the Bengali language, sahajBERT1 is a collaborative pre-trained ALBERT model that leverages these innovations. It utilizes masked language modeling (MLM) and Sentence Order Prediction (SOP) objectives to effectively learn from and understand Bengali text. This approach enables sahajBERT to perform various natural language processing tasks with high accuracy and efficiency, making it a powerful tool for Bengali language applications.

Table 3. Architectural and Training Objective Details of Various Pretrained Language Models

Table 3. Architectural and Training Objective Details of Various Pretrained Language Models

Architecture	Model	Layers	Attention Heads	Parameters	Objective Type During Training	Embedding Size
ELECTRA	BanglaBERT	12	12	110M	MLM with Replaced Token Detection (RTD)	768
BERT	BanglaBERT Base	12	12	110M	Masked Language Model (MLM)	768
	mBERT	12	12	110M	Multilingual Masked Language Model (MLM)	768
	XLM-RoBERTa	24	16	125M	Masked Language Model (MLM)	768
ALBERT	sahajBERT	24	16	18M	Multilingual Masked Language Model (MLM)	128

4. Dataset Description

5. Implementation Details

5.2. Experiment 2: Bangla Hate Speech Detection Using LLMs

5.2.1. Data Selection

For our analysis, we randomly selected 400 data points from each of the "Dataset 1," "Dataset 2," and "Dataset 3" to assess the effectiveness of Zero Shot and Few Shot prompts. Specifically, we focused on data instances where the comments contained more than 5 words. Each label category, including Hate Speech (HS) and Non-Hate Speech (NH), comprises 100 instances from each dataset. In the case of "Dataset 2," which contains multi-label categories (Personal Hate, Political Hate, Religious Hate, Geopolitical Hate, and Gender Abusive Hate), we ensured that the selected samples represented a balanced mix of these categories. These samples were extracted from the training subsets of the respective datasets. Considering the cost implications associated with using the OpenAI GPT-3.5 Turbo API and Gemini 1.5 Pro API, we opted to limit our experiments to this subset of the dataset rather than the entire corpus.

5.2.2. Prompting Template

a)

Zero-shot prompting for Bangla Hate Speech Detection

Zero-shot prompting for Bangla Hate Speech Detection involves guiding a model to classify text as either Hate Speech or Non-Hate Speech without prior training on such labels. In this context, hate speech includes offensive or derogatory language directed at individuals or groups, while non-hate speech conveys neutral or positive content. The model’s objective is to accurately classify the text based solely on its content.

Figure 7 illustrates the application of zero-shot prompting for Bangla Hate Speech Detection using the "Dataset 1" dataset. The dataset consists of premise-text pairs labeled as either Hate Speech or Non-Hate Speech. The figure compares the performance of two models: GPT-3.5 Turbo and Gemini 1.5 Pro. Both models classify the text based solely on its content without prior training on the specific labels in the dataset.

Figure 9 focuses on the multi-label prompting approach for the "Dataset 2" dataset, guiding the model to predict one or more specific hate speech categories associated with a given text. The categories include Personal Hate, Political Hate, Religious Hate, Geopolitical Hate, and Gender Abusive Hate.

Figure 11 Illustrates the application of Zero-shot prompting in Bangla Hate Speech Detection using "Dataset 3". Similar to Table 6 and Table 7, this Table 8 demonstrates how the model can classify text accurately as hate speech (’1’) or non-hate (’0’) without any training data. It highlights the model’s ability to generalize from pre-existing knowledge to classify hate speech in Bengali texts.

b)

Few-shot prompting for Bangla Hate Speech Detection:

Few-shot prompting in Bangla Hate Speech Detection involves providing the model with a limited number of example texts labeled as Hate Speech or Non-Hate Speech to guide its predictions regarding the content expressed in subsequent texts. The model aims to determine the hate speech label associated with each text based on the provided examples, without extensive training on hate speech-labeled data. For instance, in a 5-shot scenario, the model receives five labeled examples for training before making predictions on new data, while in a 10-shot scenario, it receives ten examples, and so forth. In a 15-shot scenario, the model receives fifteen labeled examples, enhancing its ability to understand and classify the content expressed in the text accurately. The model utilizes the provided examples to inform its predictions, improving its classification performance as the number of examples increases.

Figure 8 showcases the application of Few-shot prompting in Bangla Hate Speech Detection using "Dataset 1". This Table 10, Table 11 and Table 12 presents the labeled text examples used for training the model with a limited number of example texts labeled as Hate Speech (HS) or Non-Hate Speech (NH). The model, without extensive training on hate speech-labeled data, accurately predicts the hate speech label associated with each text based on the provided examples.

Figure 10 demonstrates the Few-shot capabilities of Gemini Pro in Bangla Hate Speech Detection using "Dataset 2". It illustrates how Gemini Pro effectively makes accurate predictions with a small number of training instances. The Table 10, Table 11 and Table 12 showcases the model’s ability in a multi-label few-shot prompting scenario, where it is trained on a limited number of example texts. Each example is labeled with one or more hate speech categories, including Personal Hate, Political Hate, Religious Hate, Geopolitical Hate, and Gender Abusive Hate. Gemini Pro uses this training to accurately predict and classify hate speech categories in subsequent texts, showcasing its robustness in handling diverse forms of hate speech in Bengali language.

Figure 12 illustrates the application of Few-shot prompting in Bangla Hate Speech Detection using "Dataset 3". This figure demonstrates how the model learns to classify text accurately as hate speech (’1’) or non-hate (’0’) with minimal training data. The Table 10, Table 11 and Table 12 showcases the model’s ability in a multi-label few-shot prompting scenario. It highlights the model’s ability to generalize from a small number of labeled examples to classify hate speech in Bengali texts.

Figure 7. Illustration of Prompt Design for Zero-Shot Learning with Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 1

Figure 7. Illustration of Prompt Design for Zero-Shot Learning with Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 1

Figure 8. Visual Representation of Prompt Design for Few-Shot Learning using Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 1

Figure 8. Visual Representation of Prompt Design for Few-Shot Learning using Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 1

Figure 9. Illustration of Prompt Design for Zero-Shot Learning with Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 2

Figure 9. Illustration of Prompt Design for Zero-Shot Learning with Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 2

Figure 10. Visual Representation of Prompt Design for Few-Shot Learning using Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 2

Figure 10. Visual Representation of Prompt Design for Few-Shot Learning using Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 2

Figure 11. Illustration of Prompt Design for Zero-Shot Learning with Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 3

Figure 11. Illustration of Prompt Design for Zero-Shot Learning with Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 3

Figure 12. Visual Representation of Prompt Design for Few-Shot Learning using Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 3

Figure 12. Visual Representation of Prompt Design for Few-Shot Learning using Gemini 1.5 Pro and GPT-3.5 Turbo in Dataset 3

5.2.3. Control Parameters for Large Language Models

This Table 5 summarizes the key control parameters and their values tailored for Bangla hate speech detection, ensuring unbiased model behavior in generating responses.

5.2.4. Performance Evaluation of LLMs

To objectively assess the performance of GPT-3.5 Turbo and Gemini 1.5 Pro in detecting Bangla hate speech, we use metrics such as Accuracy, Precision, Recall, and F1 score. These metrics provide a comprehensive understanding of each model’s strengths and weaknesses. Table 9, Table 10, Table 11 and Table 12 summarizes the performance metrics for both models, offering insights into their proficiency in handling the specific task of Bangla hate speech detection.

5.2.5. Error Analysis for LLMs

Error analysis is crucial in identifying and understanding the types and sources of errors made by LLMs in Bangla hate speech detection. By analyzing these errors, we can gain insights into the model’s weaknesses and improve its performance. Figure 5 illustrates the common error types encountered in this task, providing a detailed breakdown of the misclassifications.

6. Result Analysis

6.1. Quantitative Analysis

Table 6 demonstrates that BanglaBERT demonstrates the highest performance across all metrics, with an accuracy of 92.25%, precision of 92.23%, recall of 92.27%, and an F1-score of 92.19%. This indicates its strong performance in hate speech detection. Bangla BERT Base performs slightly lower, with accuracy, precision, recall, and F1-score around 91.29%, 91.30%, 91.24%, and 91.27%, respectively, showing it is a strong model, though not as effective as BanglaBERT. mBERT has an accuracy of 91.28% and precision of 91.30%, but it excels in recall (92.24%) and F1-score (92.19%), making it comparable to BanglaBERT. XLM-RoBERTa shows an accuracy of 91.22% and precision of 91.36%, but its F1-score drops to 90.27%, indicating a slight trade-off between precision and recall. sahajBERT has the lowest performance among the evaluated models, with an accuracy of 90.67%, precision of 90.88%, recall of 90.14%, and an F1-score of 90.39%. Despite performing well, sahajBERT is less effective compared to the other models listed. This analysis highlights BanglaBERT as the top-performing model for the BD-SHS dataset, followed closely by mBERT due to its high recall and F1-score.

Table 6. Performance of Pretrained Language Models on the BD-SHS Dataset

Table 6. Performance of Pretrained Language Models on the BD-SHS Dataset

Model	Accuracy	Precision	Recall	F1-Score
BanglaBERT	0.9225	0.9223	0.9227	0.9219
Bangla BERT Base	0.9129	0.9130	0.9124	0.9127
mBERT	0.9128	0.9130	0.9224	0.9219
XLM-RoBERTa	0.9122	0.9136	0.9128	0.9027
sahajBERT	0.9067	0.9088	0.9014	0.9039

Table 7 shows that BanglaBERT emerges as the top performer in Bengali hate speech detection, excelling across several key metrics. It achieves the highest accuracy at 89.21%, indicating that it correctly classifies 89.21% of the samples. Its recall is equally impressive at 89.21%, demonstrating its effectiveness in identifying actual positive samples. BanglaBERT also leads with an F1-Score of 89.20%, reflecting a balanced performance between precision and recall. However, its precision is slightly lower at 88.05%, which means that while it identifies most positive samples, a small proportion of its positive predictions are incorrect. In close competition, Bangla BERT Base exhibits an accuracy of 88.53% and shines with the highest precision among the models at 89.03%. This indicates that it has a high ratio of true positive predictions to total predicted positives. Its recall and F1-Score are 88.53% and 88.49%, respectively, showcasing its reliability and balanced performance, though marginally behind BanglaBERT in recall and F1-Score. Both mBERT and sahajBERT present similar results, each attaining an accuracy of 87.93%. Their precision and F1-Scores are closely matched, with mBERT achieving a precision of 88.14% and an F1-Score of 87.92%, while sahajBERT scores 88.21% in precision and 87.91% in F1-Score. These results suggest that both models are competent, with minor variations in their ability to balance precision and recall. XLM-RoBERTa, while still competitive, ranks lowest among the evaluated models. It achieves an accuracy of 87.23%, precision of 87.32%, recall of 87.23%, and an F1-Score of 87.23%. Despite being at the lower end of the performance spectrum in this comparison, XLM-RoBERTa still offers a robust performance, underscoring the overall competitive nature of these models in handling Bengali hate speech detection.

Table 7. Performance of Pretrained Language Models on the Bengali Hate Speech Dataset v1.0 & v2.0

Table 7. Performance of Pretrained Language Models on the Bengali Hate Speech Dataset v1.0 & v2.0

Model	Accuracy	Precision	Recall	F1-Score
BanglaBERT	0.8921	0.8814	0.8921	0.8920
Bangla BERT Base	0.8853	0.8903	0.8853	0.8849
mBERT	0.8793	0.8805	0.8793	0.8792
XLM-RoBERTa	0.8723	0.8732	0.8723	0.8723
sahajBERT	0.8793	0.8821	0.8793	0.8791

Table 8 provides a comprehensive evaluation of several models on the Bengali Hate Dataset, revealing that Bangla BERT Base achieved the highest accuracy at 91.34%, indicating it correctly classified approximately 91.34% of the instances. Following closely, BanglaBERT and mBERT performed well with accuracies of 90.42% and 90.21%, respectively, while sahajBERT and XLM-RoBERTa had lower accuracies of 85.63% and 85.52%. In terms of precision, mBERT stands out with the highest value of 91.43%, suggesting a high rate of correctly identified positive instances, followed by Bangla BERT Base and BanglaBERT with similar high precision values of 91.76% and 90.87%. SahajBERT and XLM-RoBERTa, however, have lower precision values of 78.07% and 77.68%, indicating more false positives. Bangla BERT Base again leads with a recall of 91.12%, closely followed by BanglaBERT at 90.25% and mBERT at 90.84%, whereas sahajBERT and XLM-RoBERTa have lower recall values of 84.81% and 81.84%, respectively, indicating they miss more positive instances. The highest F1-Score is achieved by Bangla BERT Base at 91.54%, reflecting a strong balance between precision and recall, with BanglaBERT and mBERT also performing well with F1-Scores of 90.63% and 91.26%, respectively. Conversely, sahajBERT and XLM-RoBERTa have lower F1-Scores of 80.14% and 78.92%, reflecting their lower precision and recall. Overall, Bangla BERT Base demonstrates the best performance across all metrics, making it the most effective model for the Bengali Hate Dataset, while BanglaBERT and mBERT also show strong performance, particularly in precision and recall, making them reliable choices for hate speech detection. In contrast, sahajBERT and XLM-RoBERTa show comparatively lower performance across all metrics, suggesting they are more prone to false positives and false negatives, respectively.

Table 8. Performance of Pretrained Language Models on the Bengali Hate Dataset

Table 8. Performance of Pretrained Language Models on the Bengali Hate Dataset

Model	Accuracy	Precision	Recall	F1-Score
BanglaBERT	0.9042	0.9087	0.9025	0.9063
Bangla BERT Base	0.9134	0.9176	0.9112	0.9154
mBERT	0.9021	0.9143	0.9084	0.9126
XLM-RoBERTa	0.8552	0.7768	0.8184	0.7892
sahajBERT	0.8563	0.7807	0.8481	0.8014

This Table 9 presents a comparative analysis of the performance metrics of two language models, GPT 3.5 Turbo and Gemini 1.5 Pro, across three distinct datasets in a zero-shot learning setting. In Dataset 1, GPT 3.5 Turbo achieves an accuracy of 86.61%, with precision, recall, and F1-score values closely aligned at 86.69%, 86.71%, and 86.65%, respectively, while Gemini 1.5 Pro achieves 82.20% accuracy, with precision, recall, and F1-score values around 82.18%, 82.24%, and 82.19%, respectively. Moving to Dataset 2, GPT 3.5 Turbo demonstrates an accuracy of 80.29%, with precision, recall, and F1-score values of approximately 80.31%, 80.24%, and 80.27%, respectively, whereas Gemini 1.5 Pro shows a slightly higher accuracy of 81.30%, maintaining consistent precision, recall, and F1-score values of 81.30%. In Dataset 3, GPT 3.5 Turbo achieves an accuracy of 83.31%, with precision, recall, and F1-score values all hovering around 83.30% and 83.31%, respectively, while Gemini 1.5 Pro demonstrates superior performance with an accuracy of 87.76%, achieving precision, recall, and F1-score values of 87.82%, 87.69%, and 87.75%, respectively. Overall, both models show competitive performance metrics across datasets, with GPT 3.5 Turbo maintaining stable performance and Gemini 1.5 Pro exhibiting noticeable improvements, particularly in Dataset 3. However, it is important to note that these zero-shot results generally indicate worse performance compared to models fine-tuned on specific tasks, such as pre-trained language models, due to the lack of task-specific training and adaptation.

Table 9. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a Zero-Shot Learning Scenario

Table 9. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a Zero-Shot Learning Scenario

Dataset	Model	Accuracy	Precision	Recall	F1-Score
Dataset 1	GPT 3.5 Turbo	0.8661	0.8669	0.8671	0.8665
	Gemini 1.5 Pro	0.8220	0.8218	0.8224	0.8219
Dataset 2	GPT 3.5 Turbo	0.8029	0.8031	0.8024	0.8027
	Gemini 1.5 Pro	0.8130	0.8130	0.8130	0.8130
Dataset 3	GPT 3.5 Turbo	0.8331	0.8330	0.8331	0.8331
	Gemini 1.5 Pro	0.8776	0.8782	0.8769	0.8775

Table 10 showcases the performance of two large language models, GPT-3.5 Turbo and Gemini-1.5 Pro, across three datasets in a 5-shot learning scenario. GPT-3.5 Turbo consistently outperforms Gemini-1.5 Pro on all datasets, with the most pronounced difference in Dataset 1 (approximately 2.5 percentage points across all metrics) and the least in Dataset 2 (less than 0.2 percentage points). For Dataset 1, GPT-3.5 Turbo achieved 93.79% Accuracy, 93.85% Precision, 93.73% Recall, and 93.79% F1-Score, while Gemini-1.5 Pro scored 91.29%, 91.30%, 91.24%, and 91.27%, respectively. On Dataset 2, GPT-3.5 Turbo’s metrics remained strong and consistent with those of Dataset 1, whereas Gemini-1.5 Pro improved significantly to 93.65% Accuracy, 93.71% Precision, 93.79% Recall, and 93.64% F1-Score. For Dataset 3, GPT-3.5 Turbo demonstrated its best performance with metrics around 94.65%, compared to Gemini-1.5 Pro’s 92.29% Accuracy, 92.30% Precision, 92.24% Recall, and 92.27% F1-Score. Overall, GPT-3.5 Turbo showed higher consistency and robustness across datasets, while Gemini-1.5 Pro exhibited more variation, indicating potential sensitivity to dataset characteristics. Notably, the 5-shot learning approach consistently outperforms both zero-shot and pretrained language models due to its ability to leverage a small amount of task-specific training data, allowing for improved adaptation to the task at hand.

Table 10. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a 5-Shot Learning Scenario

Table 10. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a 5-Shot Learning Scenario

Dataset	Model	Accuracy	Precision	Recall	F1-Score
Dataset 1	GPT 3.5 Turbo	0.9379	0.9385	0.9373	0.9379
	Gemini 1.5 Pro	0.9129	0.9130	0.9124	0.9127
Dataset 2	GPT 3.5 Turbo	0.9378	0.9382	0.9374	0.9378
	Gemini 1.5 Pro	0.9365	0.9371	0.9379	0.9364
Dataset 3	GPT 3.5 Turbo	0.9465	0.9463	0.9467	0.9465
	Gemini 1.5 Pro	0.9229	0.9230	0.9224	0.9227

Table 11 provides a detailed performance comparison of two advanced large language models, GPT-3.5 Turbo and Gemini 1.5 Pro, across three different datasets in a 10-shot learning scenario using four key evaluation metrics: Accuracy, Precision, Recall, and F1-Score. For Dataset 1, GPT-3.5 Turbo demonstrates strong performance with an accuracy of 94.53%, precision of 94.48%, recall of 94.57%, and F1-Score of 94.52%, outperforming Gemini 1.5 Pro which has an accuracy of 93.75%, precision of 93.72%, recall of 93.78%, and F1-Score of 93.76%. On Dataset 2, GPT-3.5 Turbo maintains high performance with an accuracy of 95.67%, precision of 95.63%, recall of 95.69%, and F1-Score of 95.66%, but is surpassed by Gemini 1.5 Pro, which achieves an accuracy of 96.67%, precision of 96.63%, recall of 96.69%, and F1-Score of 96.66%. For Dataset 3, GPT-3.5 Turbo again shows strong performance with an accuracy of 95.67%, precision of 95.63%, recall of 95.69%, and F1-Score of 95.66%, whereas Gemini 1.5 Pro performs less well with an accuracy of 93.20%, precision of 93.18%, recall of 93.24%, and F1-Score of 93.19%. Overall, GPT-3.5 Turbo generally outperforms Gemini 1.5 Pro on Datasets 1 and 3, while Gemini 1.5 Pro shows superior performance on Dataset 2, with trends in precision, recall, and F1-Score following the accuracy trends. The 10-shot learning approach consistently demonstrates better performance compared to 5-shot learning, zero-shot, and pretrained language models. This improvement is attributed to the increased amount of task-specific training data, allowing the models to better adapt and generalize to the evaluation tasks, resulting in higher accuracy and more balanced precision-recall trade-offs.

Table 11. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a 10-Shot Learning Scenario

Table 11. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a 10-Shot Learning Scenario

Dataset	Model	Accuracy	Precision	Recall	F1-Score
Dataset 1	GPT 3.5 Turbo	0.9453	0.9448	0.9457	0.9452
	Gemini 1.5 Pro	0.9375	0.9372	0.9378	0.9376
Dataset 2	GPT 3.5 Turbo	0.9567	0.9563	0.9569	0.9566
	Gemini 1.5 Pro	0.9667	0.9663	0.9669	0.9666
Dataset 3	GPT 3.5 Turbo	0.9567	0.9563	0.9569	0.9566
	Gemini 1.5 Pro	0.9320	0.9318	0.9324	0.9319

In the comparative analysis presented in Table 12, GPT 3.5 Turbo and Gemini 1.5 Pro were evaluated across three distinct datasets in a 15-shot learning scenario. Across Dataset 1, GPT 3.5 Turbo slightly outperformed Gemini 1.5 Pro with higher accuracy (97.33% compared to 97.11%), precision (97.31% compared to 97.02%), recall (97.35% compared to 97.15%), and F1-score (97.33% compared to 97.13%). Moving to Dataset 2 and Dataset 3, GPT 3.5 Turbo consistently demonstrated superior performance with noticeably higher accuracy, precision, recall, and F1-scores compared to Gemini 1.5 Pro. Specifically, in Dataset 2, GPT 3.5 Turbo achieved an accuracy and F1-score of 98.42%, while Gemini 1.5 Pro scored 97.23% and 97.23% respectively. In Dataset 3, GPT 3.5 Turbo maintained high metrics with 98.53% accuracy and 98.53% F1-score, whereas Gemini 1.5 Pro achieved 97.47% and 97.48%. This comprehensive analysis highlights GPT 3.5 Turbo’s consistent superiority over Gemini 1.5 Pro across diverse datasets in the 15-shot learning scenario. The 15-shot learning approach demonstrates superior performance compared to 5-shot and 10-shot learning methods, as well as zero-shot and pretrained language models. This improvement can be attributed to the increased availability of task-specific training data, allowing the models to refine their understanding and optimization for the evaluation tasks, resulting in higher accuracy and precision-recall balance. Additionally, the 15-shot learning scenario benefits from a larger sample of task-specific examples during training, facilitating deeper model adaptation and more accurate predictions.

Table 12. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a 15-Shot Learning Scenario

Table 12. Performance Comparison of GPT 3.5 Turbo and Gemini 1.5 Pro on Three Datasets in a 15-Shot Learning Scenario

Dataset	Model	Accuracy	Precision	Recall	F1-Score
Dataset 1	GPT 3.5 Turbo	0.9733	0.9731	0.9735	0.9733
	Gemini 1.5 Pro	0.9711	0.9702	0.9715	0.9713
Dataset 2	GPT 3.5 Turbo	0.9842	0.9840	0.9844	0.9842
	Gemini 1.5 Pro	0.9723	0.9727	0.9726	0.9723
Dataset 3	GPT 3.5 Turbo	0.9853	0.9851	0.9855	0.9853
	Gemini 1.5 Pro	0.9747	0.9743	0.9746	0.9748

Figure 11 illustrates an error analysis of Bangla language social media posts evaluated by PLMs, emphasizing the critical role of error analysis in evaluating model performance. Firstly, it sheds light on the model’s proficiency in interpreting nuanced language, evident in its misclassification of sentiments related to war, equality, and peace in the first post. This highlights the necessity for the model to better comprehend complex socio-political discussions for accurate sentiment analysis and contextual understanding in sensitive topics. Secondly, error analysis identifies specific challenges faced by the model, such as its difficulty in distinguishing neutral or rhetorical statements from positive sentiments, as observed in the misclassifications of the second and third posts. Understanding these challenges is pivotal for refining model training strategies to enhance its performance in real-world applications where precise sentiment classification is crucial. Moreover, correct classifications, exemplified by the fourth and sixth posts, validate the model’s capability to accurately interpret sentiments concerning human rights issues and neutral content, respectively. These instances underscore areas where the model excels and provides reliable predictions, bolstering confidence in its performance. Conversely, the misclassification of the fifth post, which discusses intricate political and religious themes, exposes significant hurdles in the model’s comprehension of such culturally specific references. This insight underscores the need for targeted improvements to broaden the model’s understanding of diverse content, thereby enhancing its overall reliability in Bangla language processing tasks. Ultimately, conducting thorough error analysis yields actionable insights for enhancing PLMs’ proficiency in sentiment analysis and contextual understanding of Bangla social media discourse. By addressing identified challenges and leveraging strengths, these efforts aim to bolster the models’ accuracy and effectiveness in handling complex linguistic nuances and socio-cultural contexts.

Figure 13. Error Analysis of Pre-Trained Language Models on Bangla Hate Speech Detection.

Figure 13. Error Analysis of Pre-Trained Language Models on Bangla Hate Speech Detection.

6.3. Comparison with an Existing Approaches

Table 13. Detailed Comparison of Approaches from Selected Research Papers in Bangla Hate Speech Detection

Table 13. Detailed Comparison of Approaches from Selected Research Papers in Bangla Hate Speech Detection

Paper	Dataset	Approach	Performance Metrics	Comments
This paper	BD-SHS, Bengali Hate Speech Dataset v1.0, Bengali Hate Speech Dataset v2.0, Bengali Hate Dataset	In the context of 15-shot learning, both GPT 3.5 Turbo and Gemini 1.5 Pro were evaluated.	97.33% in Dataset 1, 98.42% in Dataset 2, and 98.53% in Dataset 3.	GPT 3.5 Turbo excelled particularly in Dataset 1, Dataset 2 and Dataset 3, demonstrating significantly higher accuracy compared to Gemini 1.5 Pro.
Saroar et al. [2]	Offensive posts filtered: 8.5k. Non-offensive posts identified: 8.5k. Final manually labeled dataset: 15k posts (balanced with 7.5k offensive and 7.5k non-offensive posts).	The existing BanglaBERT model, pre-trained on 18.6 GB of Bengali text (1 million steps over 3 billion tokens), was retrained with 1.5 million offensive posts for 15 epochs (almost 2 million steps) in batches of 64 samples using MLM and the Adam optimizer with a learning rate of 5e-5.	Bangla Hate BERT: Accuracy - 94.3%, F1 Score - 94.1%	The dataset is balanced with equal offensive and non-offensive posts, and high-quality labels from manual annotation. A limitation is the need for a large corpus for traditional models. However, LLMs can generalize from large-scale pre-existing datasets, reducing the need for extensive domain-specific annotated data.
Rezaul et al. [15]	The dataset has 100,000 annotated hate speech statements, covering political, personal, gender-based, geopolitical, and religious hate, created with a bootstrapping and semi-automatic annotation approach.	The MC-LSTM integrates BengFastText embeddings for hate speech detection, capturing contextual and semantic information from Bengali texts. Additionally, traditional ML models (SVM, KNN, LR, NB, DT, RF, GBT) and embedding models (Word2Vec, GloVe) were trained for a comprehensive performance comparison.	Achieved up to 90.45% F1-score.	The authors’ traditional model training approach didn’t address the need for a large corpus. LLMs mitigate this by generalizing from large pre-existing datasets, showing that LLMs offer a more efficient and adaptive alternative to traditional methods.
Nauros et al. [4]	BD-SHS, the largest Bangla hate speech dataset, consists of 50,281 comments manually labeled in different social contexts. 24,156 comments are tagged as hate speech (HS).	Various ML models, including SVM and Bi-LSTM, were used to identify and categorize hate speech, combined with word embeddings like pre-trained formal (BFT, MFT) and informal (IFT) embeddings.	Weighted F1-score of 91.00%	LLMs can be leveraged to mitigate the need for extensive labeled data, which is often time-consuming to gather, by utilizing few-shot learning techniques and transfer learning to achieve robust performance with minimal annotated examples.
Jobair et al. [3]	The new dataset consists of 8,600 user comments from Facebook and YouTube, categorized into sports, religion, politics, entertainment, and others.	Conducted a comprehensive study using five distinct models to analyze abusive language in Bengali. The models tested include CNN, LSTM, Bi-LSTM, GRU, and BERT. Additionally, we ran these models on an existing dataset of 30,000 records to compare performance across different datasets.	The BERT model outperformed others with a 97% accuracy and an F1-score of 96%.	LLMs can effectively minimize the reliance on extensive labeled datasets. Leveraging techniques like few-shot learning, zero-shot learning, and transfer learning, LLMs achieve robust performance even with minimal annotated examples, circumventing the time-consuming process of gathering extensive labeled data.

7. Limitations

8. Future Research Directions

9. Conclusion

References

1. Das, A. K., Al Asif, A., Paul, A., & Hossain, M. N. (2021). Bangla hate speech detection on social media using attention-based recurrent neural network. Journal of Intelligent Systems, 30(1), 578-591. [CrossRef]
2. Md Saroar Jahan, Mainul Haque, Nabil Arhab, and Mourad Oussalah. 2022. BanglaHateBERT: BERT for Abusive Language Detection in Bengali. In Proceedings of the Second International Workshop on Resources and Techniques for User Information in Abusive Language Analysis, pages 8–15, Marseille, France. European Language Resources Association.
3. Jobair, Md & Das, Dhrubajyoti & Islam, Binte & Dhar, Munna. (2023). Bengali Hate Speech Detection with BERT and Deep Learning Models. [CrossRef]
4. Nauros Romim, Mosahed Ahmed, Md Saiful Islam, Arnab Sen Sharma, Hriteshwar Talukder, and Mohammad Ruhul Amin. 2022. BD-SHS: A Benchmark Dataset for Learning to Detect Online Bangla Hate Speech in Different Social Contexts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5153–5162, Marseille, France. European Language Resources Association.
5. Mithun Das, Somnath Banerjee, Punyajoy Saha, and Animesh Mukherjee. 2022. Hate Speech and Offensive Language Detection in Bengali. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 286–296, Online only. Association for Computational Linguistics.
6. Kumarage, T., Bhattacharjee, A., & Garland, J. (2024). Harnessing artificial intelligence to combat online hate: Exploring the challenges and opportunities of large language models in hate speech detection. arXiv preprint arXiv:2403.08035.
7. Guo, K., Hu, A., Mu, J., Shi, Z., Zhao, Z., Vishwamitra, N., & Hu, H. (2023, December). An investigation of large language models for real-world hate speech detection. In 2023 International Conference on Machine Learning and Applications (ICMLA) (pp. 1568-1573). IEEE.
8. Sarthak Roy, Ashish Harshvardhan, Animesh Mukherjee, and Punyajoy Saha. 2023. Probing LLMs for hate speech detection: strengths and vulnerabilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6116–6128, Singapore. Association for Computational Linguistics.
9. Flor Miriam Plaza-del-arco, Debora Nozza, and Dirk Hovy. 2023. Respectful or Toxic? Using Zero-Shot Learning with Language Models to Detect Hate Speech. In The 7th Workshop on Online Abuse and Harms (WOAH), pages 60–68, Toronto, Canada. Association for Computational Linguistics.
10. Kabir, M., Islam, M. S., Laskar, M. T. R., Nayeem, M. T., Bari, M. S., & Hoque, E. (2023). Benllmeval: A comprehensive evaluation into the potentials and pitfalls of large language models on bengali nlp. arXiv preprint arXiv:2309.13173. [CrossRef]
11. Rohanian, Omid, Mohammadmahdi Nouriborji, and David A. Clifton. "Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processing." arXiv preprint arXiv:2401.00579 (2023). [CrossRef]
12. Sarker, Manash & Hossain, Md & Rahman, Liza & Sakib, Nazmus & Farooq, Abdullah. (2022). A Machine Learning Approach to Classify Anti-social Bengali Comments on Social Media. 1-6. 2022.9836407. [CrossRef]
13. Sultana, Sherin & Redoy, Md & Nahian, Jabir & Masum, Abu Kaisar Mohammad & Abujar, Sheikh. (2022). Detection of Abusive Bengali Comments for Mixed Social Media Data Using Machine Learning. [CrossRef]
14. Hossain Junaid, Md & Hossain, Faisal & Rahman, Mohammad. (2021). Bangla Hate Speech Detection in Videos Using Machine Learning. [CrossRef]
15. Karim, M. R., Chakravarthi, B. R., McCrae, J. P., & Cochez, M. (2020, October). Classification benchmarks for under-resourced bengali language based on multichannel convolutional-lstm network. In 2020 IEEE 7th international conference on Data Science and Advanced Analytics (DSAA) (pp. 390-399). IEEE.
16. Ishmam, Alvi & Sharmin, Sadia. (2019). Hateful Speech Detection in Public Facebook Pages for the Bengali Language. 555-560. [CrossRef]
17. Karim, Rezaul & Dey, Sumon & Chakravarthi, Bharathi. (2020). DeepHateExplainer: Explainable Hate Speech Detection in Under-resourced Bengali Language.
18. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. http://arxiv.org/abs/1810.04805.
19. Sarker, S. (2020). BanglaBERT: Bengali Mask Language Model for Bengali Language Understanding. https://github.com/sagorbrur/bangla-bert.
20. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [CrossRef]
21. Conneau, Alexis, et al. "Unsupervised cross-lingual representation learning at scale." arXiv preprint arXiv:1911.02116 (2019). [CrossRef]
22. Clark, K., Luong, M. T., Le, Q. V., & Manning, C. D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [CrossRef]
23. Bhattacharjee, A., Hasan, T., Ahmad, W.U., Samin, K., Islam, M.S., Iqbal, A., Rahman, M.S. and Shahriyar, R., 2021. BanglaBERT: Language model pretraining and benchmarks for low-resource language understanding evaluation in Bangla. arXiv preprint arXiv:2101.00204. [CrossRef]
24. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
25. Bhavish Pahwa and Bhavika Pahwa. 2023. BpHigh at SemEval-2023 Task 7: Can Fine-tuned Cross-encoders Outperform GPT-3.5 in NLI Tasks on Clinical Trial Data?. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1936–1944, Toronto, Canada. Association for Computational Linguistics.
26. Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., ... & Huang, X. (2023). A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420.
27. Sokolova, Marina & Japkowicz, Nathalie & Szpakowicz, Stan. (2006). Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation. AI 2006: Advances in Artificial Intelligence, Lecture Notes in Computer Science. Vol. 4304. 1015-1021. [CrossRef]
28. Goutte, C., Gaussier, E. (2005). A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In: Losada, D.E., Fernández-Luna, J.M. (eds) Advances in Information Retrieval. ECIR 2005. Lecture Notes in Computer Science, vol 3408. Springer, Berlin, Heidelberg. [CrossRef]
29. Hamdy Mubarak, Hend Al-Khalifa, and Khaloud Suliman Alkhalefah. 2024. Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8008–8015, Torino, Italia. ELRA and ICCL.
30. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., ... & Liu, T. (2023). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.