1. Introduction

2. Related Work

3. Methods

4. Experiments

4.1. Experiment Setup

4.1.1. Dataset

We chose 11 different languages available in MultiEmo Sentiment Corpus: Chinese, Dutch, English, French, German, Italian, Japanese, Polish, Portuguese, Russian, and Spanish. The MultiEmo dataset contains two types of data: text and sentence. For text type (Text): Across all domains, there are a total of 6573 texts in the training set, 823 texts in the validation set, and 820 texts in the test set, making a total of 8216 texts. For sentence type (Sentence): Across all domains, there are a total of 45974 sentences in the training set, 5745 sentences in the validation set, and 5747 sentences in the test set, making a total of 57466 sentences. As show in Table 1, etc.

We analyze the average length of text and sentence data.

Text Length:

• The ’Medicine’ domain has the longest average text length at 782 units.
• The ’School’ domain has the shortest average text length at 427 units.
• The average text length for ’Hotels’ and ’Products’ is relatively similar, at 773 and 756 units respectively.

Figure 3. The average length of text and sentence across different domains.

Figure 3. The average length of text and sentence across different domains.

Sentence Length:

• The ’School’ and ’Medicine’ domains have the longest average sentence lengths, at 110 and 111 units respectively.
• The ’Hotels’ domain has the shortest average sentence length at 92 units.
• The ’Products’ domain has an average sentence length of 98 units, which is in the mid-range.

Text vs Sentence Length:

• Across all domains, the average text length is significantly longer than the average sentence length. This is expected as a text usually includes multiple sentences.
• The difference between average text length and sentence length is greatest in the ’School’ domain, suggesting that texts in this domain may contain more sentences on average compared to the other domains.
• The ’Medicine’ domain, despite having the longest average text length, does not have the largest difference between text and sentence length. This suggests that while the texts are long, they may not contain as many sentences as in the ’School’ domain but rather longer sentences.

4.1.2. Baseline

We compare five strong multilingual sentiment analysis baselines, including:

• LASER + BiLSTM [32] presents a system for learning sentence embeddings across 93 languages using a shared encoder, enabling zero-shot transfer of NLP models to new languages and introducing a new multilingual test set.
• MultiFit [24] proposes an efficient method for fine-tuning language models in multiple languages, especially useful for low-resource languages, and includes a zero-shot approach using existing cross-lingual models.
• XLM-RoBERTa [25] introduces XLM-RoBERTa, a Transformer-based model pre-trained on 100 languages, showing significant performance gains in cross-lingual tasks and a detailed analysis of scaling effects on model performance.
• LLaMa3_zero-shot and LLaMa3_few-shot) [31] are developed by Meta AI, capable of generating text in response to instructions, demonstrating strong understanding and generation capabilities for complex language tasks.

4.1.3. Experiment Result for Hyperparameter Tuning

In this set of experiments, we conducted the experiments separately based on languages. The pre-trained model we used was Meta’s Llama-3-8B-Instruct, which we fine-tuned efficiently using the LoRA technique, with a learning rate of 1.0e-4, a warmup_ratio of 0.1, a maximum sequence length of 4096, a temperature of 0.1, and a top_p of 0.9. We ran 5 training epochs to train the models, and all experiments were carried out on an NVIDIA A100 GPU with a memory capacity of 40GB.

4.1.4. Evaluation

Following previous work [33], We leverage Accuracy and F1 Score as the evaluation metrics. Accuracy is defined as the proportion of correctly classified samples out of the total samples and is calculated as:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$$

(1)

where:

• TP (True Positive) represents the number of true positive instances,
• TN (True Negative) represents the number of true negative instances,
• FP (False Positive) represents the number of false positive instances, and
• FN (False Negative) represents the number of false negative instances.

F1 Score combines the model’s precision and recall and is calculated as:

$$F1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(2)

where:

• $\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$ is the precision of the model,
• $\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$ is the recall of the model.

5. Discussion

6. Conclusions

Abbreviations

References

1. Marvin M. Agüero-Torales, José I. Abreu Salas, Antonio G. López-Herrera. Deep learning and multilingual sentiment analysis on social media data: An overview. Applied Soft Computing 2021, 107, 142–149.
2. Filip, T., Pavlíček, M., Sosík, P. (2024). Fine-tuning multilingual language models in Twitter/X sentiment analysis: a study on Eastern-European V4 languages. arXiv preprint arXiv:2408.02044.
3. Wong, E. P., M’hiri, F. (2024). Analyzing Language Bias Between French and English in Conventional Multilingual Sentiment Analysis Models. arXiv preprint arXiv:2405.06692.
4. Thakkar, G., Hakimov, S., Tadić, M. (2024). M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets. arXiv preprint arXiv:2404.01753.
5. Hasan, M. A. (2024). Ensemble Language Models for Multilingual Sentiment Analysis. arXiv preprint arXiv:2403.06060.
6. Koto, F., Beck, T., Talat, Z., Gurevych, I., Baldwin, T. (2024). Zero-shot sentiment analysis in low-resource languages using a multilingual sentiment lexicon. arXiv preprint arXiv:2402.02113.
7. Buscemi, A., Proverbio, D. (2024). Chatgpt vs gemini vs llama on multilingual sentiment analysis. arXiv preprint arXiv:2402.01715.
8. Hu, D., Wei, L., Liu, Y., Zhou, W., Hu, S. (2023). UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysis. arXiv preprint arXiv:2306.01093.
9. Wang, M., Adel, H., Lange, L., Strötgen, J., Schütze, H. (2023). Nlnde at semeval-2023 task 12: Adaptive pretraining and source language selection for low-resource multilingual sentiment analysis. arXiv preprint arXiv:2305.00090.
10. Manuvie, R., Chatterjee, S. (2023). Automated Sentiment and Hate Speech Analysis of Facebook Data by Employing Multilingual Transformer Models. arXiv preprint arXiv:2301.13668.
11. Câmara, A., Taneja, N., Azad, T., Allaway, E., Zemel, R. (2022). Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. arXiv preprint arXiv:2204.03558.
12. Antypas, D., Preece, A., Camacho-Collados, J. (2023). Negativity spreads faster: A large-scale multilingual twitter analysis on the role of sentiment in political communication.Online Social Networks and Media 2023, 33.
13. Muhammad, S. H., Adelani, D. I., Ruder, S., Ahmad, I. S., Abdulmumin, I., Bello, B. S., ... Brazdil, P. (2022). Naijasenti: A nigerian twitter sentiment corpus for multilingual sentiment analysis. arXiv preprint arXiv:2201.08277.
14. Rodrigues, R. C., Inuzuka, M. A., Gomes, J. R. S. A., Rocha, A. S., Calixto, I., & Nascimento, H. A. D. D. (2021). Zero-shot hashtag segmentation for multilingual sentiment analysis. arXiv preprint arXiv:2112.03213.
15. Barbieri, F., Anke, L. E., Camacho-Collados, J. (2021). XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. arXiv preprint arXiv:2104.12250.
16. Azhar, A. N., Khodra, M. L. (2020, September). Fine-tuning pretrained multilingual BERT model for Indonesian aspect-based sentiment analysis. In 2020 7th International Conference on Advance Informatics: Concepts, Theory and Applications (ICAICTA). IEEE 2020, 1–6.
17. Öhman, E., Pàmies, M., Kajava, K., Tiedemann, J. (2020). XED: A multilingual dataset for sentiment analysis and emotion detection. arXiv preprint arXiv:2011.01612.
18. Barriere, V., Balahur, A. (2020). Improving sentiment analysis over non-English tweets using multilingual transformers and automatic translation for data-augmentation. arXiv preprint arXiv:2010.03486.
19. Shah, S. R., Kaushik, A. (2019). Sentiment analysis on indian indigenous languages: a review on multilingual opinion mining. arXiv preprint arXiv:1911.12848.
20. Graff, M., Miranda-Jimenez, S., Tellez, E. S., Moctezuma, D. (2020). Evomsa: A multilingual evolutionary approach for sentiment analysis [application notes]. IEEE Computational Intelligence Magazine2020, 15(1), 76-88.
21. Can, E. F., Ezen-Can, A., Can, F. (2018). Multilingual sentiment analysis: An RNN-based framework for limited data. arXiv preprint arXiv:1806.04511.
22. Lu, Y., Mori, T. (2017). Deep learning paradigm with transformed monolingual word embeddings for multilingual sentiment analysis. arXiv preprint arXiv:1710.03203.
23. Ruder, S., Ghaffari, P., Breslin, J. G. (2016). Insight-1 at semeval-2016 task 5: Deep learning for multilingual aspect-based sentiment analysis. arXiv preprint arXiv:1609.02748.
24. Eisenschlos, J., Ruder, S., Czapla, P., Kadras, M., Gugger, S., Howard, J.: MultiFit: Efficient multi-lingual language model fine-tuning. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5706-5711 (2019).
25. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
26. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... McGrew, B. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
27. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A Robustly Optimized BERT Pre-training Approach with Post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China.
28. Floridi, L., Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30, 681-694.
29. Vaswani, A. (2017). Attention is all you need. arXiv preprint arXiv:1706.03762.
30. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
31. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., ... Ganapathy, R. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
32. Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597–610.
33. Kocoń, J., Miłkowski, P., Kanclerz, K. (2021, June). Multiemo: Multilingual, multilevel, multidomain sentiment analysis corpus of consumer reviews. In International Conference on Computational Science (pp. 297-312). Cham: Springer International Publishing.