Version 1
: Received: 24 September 2024 / Approved: 25 September 2024 / Online: 26 September 2024 (16:55:15 CEST)
How to cite:
Sterling, L.; Vale, K.; Martinez, A. Fusion of Visual and Textual Data for Enhanced Semantic Representations. Preprints2024, 2024092066. https://doi.org/10.20944/preprints202409.2066.v1
Sterling, L.; Vale, K.; Martinez, A. Fusion of Visual and Textual Data for Enhanced Semantic Representations. Preprints 2024, 2024092066. https://doi.org/10.20944/preprints202409.2066.v1
Sterling, L.; Vale, K.; Martinez, A. Fusion of Visual and Textual Data for Enhanced Semantic Representations. Preprints2024, 2024092066. https://doi.org/10.20944/preprints202409.2066.v1
APA Style
Sterling, L., Vale, K., & Martinez, A. (2024). Fusion of Visual and Textual Data for Enhanced Semantic Representations. Preprints. https://doi.org/10.20944/preprints202409.2066.v1
Chicago/Turabian Style
Sterling, L., Kairos Vale and Ava Martinez. 2024 "Fusion of Visual and Textual Data for Enhanced Semantic Representations" Preprints. https://doi.org/10.20944/preprints202409.2066.v1
Abstract
Generic text embeddings have demonstrated considerable success across a multitude of applications. However, these embeddings are typically derived by modeling the co-occurrence patterns within text-only corpora, which can limit their ability to generalize effectively across diverse contexts. In this study, we investigate methodologies that incorporate visual information into textual representations to overcome these limitations. Through extensive ablation studies, we introduce a novel and straightforward architecture named VisualText Fusion Network (VTFN). This architecture not only surpasses existing multimodal approaches on a range of well-established benchmark datasets but also achieves state-of-the-art performance on image-related textual datasets while utilizing significantly less training data. Our findings underscore the potential of integrating visual modalities to substantially enhance the robustness and applicability of text embeddings, paving the way for more nuanced and contextually rich semantic representations.
Keywords
Multimodal Integration; Semantic Embeddings; Representation Learning; Transfer Learning
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
