Version 1
: Received: 23 September 2024 / Approved: 24 September 2024 / Online: 24 September 2024 (12:32:35 CEST)
How to cite:
Smith, O.; Martinez, A.; Brown, N. Multimodal Information Fusion with Neural Gating. Preprints2024, 2024091917. https://doi.org/10.20944/preprints202409.1917.v1
Smith, O.; Martinez, A.; Brown, N. Multimodal Information Fusion with Neural Gating. Preprints 2024, 2024091917. https://doi.org/10.20944/preprints202409.1917.v1
Smith, O.; Martinez, A.; Brown, N. Multimodal Information Fusion with Neural Gating. Preprints2024, 2024091917. https://doi.org/10.20944/preprints202409.1917.v1
APA Style
Smith, O., Martinez, A., & Brown, N. (2024). Multimodal Information Fusion with Neural Gating. Preprints. https://doi.org/10.20944/preprints202409.1917.v1
Chicago/Turabian Style
Smith, O., Ava Martinez and Noah Brown. 2024 "Multimodal Information Fusion with Neural Gating" Preprints. https://doi.org/10.20944/preprints202409.1917.v1
Abstract
In this study, we introduce an innovative framework for multimodal learning that leverages enhanced fusion gate units within gated neural network architectures. The proposed Fusion Gate Unit (FGU) serves as a pivotal component in neural network designs, aiming to derive a comprehensive intermediate representation by amalgamating data from diverse modalities. The FGU is adept at determining the extent to which each modality influences the unit's activation through the utilization of multiplicative gating mechanisms. We conducted evaluations on a multilabel genre classification task for movies, utilizing both plot summaries and poster images as input modalities. The results demonstrate that the FGU significantly elevates the macro F-score compared to single-modality approaches and surpasses existing fusion techniques, including mixture of experts models. Additionally, we present the MM-IMDb dataset alongside this publication, which, to our knowledge, represents the most extensive publicly accessible multimodal dataset for movie genre prediction to date. This dataset is expected to facilitate further research and development in the field of multimodal information processing.
Keywords
multimodal learning; neural networks; genre classification; information fusion; gated mechanisms
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
