Text-image semantic relevance identification for aspect-based multimodal sentiment analysis

View article
PeerJ Computer Science

Main article text

 

Introduction

Methodology

Overview

Unimodal feature extraction module

Aspect and text representation

Image representation

Multimodal feature relevance identification module

Text-image cross-modal interaction layer

Image gate construction layer

Aspect-multimodal feature interaction module

Aspect interaction layer

Image feature auxiliary reconstruction layer

Multimodal feature fusion module

Experiment

Experimental settings

Datasets

Implementation details

Compared baselines

Experimental Results and Analysis

Ablation study

Parameter analysis

Values of epoch and batch size

Value of k

Value of λ

Error analysis

Conclusions

Supplemental Information

Additional Information and Declarations

Competing Interests

The authors declare there are no competing interests.

Author Contributions

Tianzhi Zhang conceived and designed the experiments, performed the experiments, analyzed the data, performed the computation work, prepared figures and/or tables, authored or reviewed drafts of the article, and approved the final draft.

Gang Zhou conceived and designed the experiments, authored or reviewed drafts of the article, and approved the final draft.

Jicang Lu conceived and designed the experiments, authored or reviewed drafts of the article, and approved the final draft.

Zhibo Li analyzed the data, prepared figures and/or tables, and approved the final draft.

Hao Wu performed the computation work, prepared figures and/or tables, and approved the final draft.

Shuo Liu performed the computation work, prepared figures and/or tables, and approved the final draft.

Data Availability

The following information was supplied regarding data availability:

The third-party datasets from ”Adapting BERT for Target-Oriented Multimodal Sentiment Classification” (DOI: https://doi.org/10.24963/ijcai.2019/751) are available at GitHub: https://github.com/jefferyYu/TomBERT.

The data of each tweet’s associated images is available at figshare: Zhang, Tianzhi (2024). Twitter Datasets.zip. figshare. Dataset. https://doi.org/10.6084/m9.figshare.25303591.v1.

The data of tweets is available in the Supplementary File.

Funding

This research is supported by the Science and Technology Research Program of the Department of Science and Technology of Henan Province (approval No.: 222102210081). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

1 Citation 1,197 Views 130 Downloads

Your institution may have Open Access funds available for qualifying authors. See if you qualify

Publish for free

Comment on Articles or Preprints and we'll waive your author fee
Learn more

Five new journals in Chemistry

Free to publish • Peer-reviewed • From PeerJ
Find out more