Abstract
Multimodal aspect-based sentiment classification (MASC) is an emerging task due to an increase in user-generated multimodal content on social platforms, aimed at predicting sentiment polarity toward specific aspect targets (i.e., entities or attributes explicitly mentioned in text-image pairs). Despite extensive efforts and significant achievements in existing MASC, substantial gaps remain in understanding fine-grained visual content and the cognitive rationales derived from semantic content and impressions (cognitive interpretations of emotions evoked by image content). In this study, we present Chimera: a cognitive and aesthetic sentiment causality understanding framework to derive fine-grained holistic features of aspects and infer the fundamental drivers of sentiment expression from both semantic perspectives and affective-cognitive resonance (the synergistic effect between emotional responses and cognitive interpretations). Specifically, this framework first incorporates visual patch features for patch-word alignment. Meanwhile, it extracts coarse-grained visual features (e.g., overall image representation) and fine-grained visual regions (e.g., aspect-related regions) and translates them into corresponding textual descriptions (e.g., facial, aesthetic). Finally, we leverage the sentimental causes and impressions generated by a large language model (LLM) to enhance the model's awareness of sentimental cues evoked by semantic content and affective-cognitive resonance. Experimental results on standard MASC datasets demonstrate the effectiveness of the proposed model, which also exhibits greater flexibility to MASC compared to LLMs such as GPT-4o. We have publicly released the complete implementation and dataset at this https URL
Abstract (translated)
多模态方面情感分类（MASC）是一项新兴任务，由于社交媒体平台上用户生成的多模态内容增多而产生。该任务旨在预测特定方面的目标的情感极性（即，在图文对中明确提到的实体或属性）。尽管在现有的MASC研究上已经付出了大量努力并取得了显著成果，但在理解细粒度视觉内容以及从语义内容和印象中得出的认知理由方面仍存在重大差距。 在这项研究中，我们提出了Chimera：一个认知与美学情感因果关系理解框架。该框架旨在通过语义视角和情感能动-认知共鸣（情感反应与认知解释之间的协同作用）来推断表达情感的基本驱动因素，并从细粒度的整体特征角度来解析方面。具体而言，此框架首先将视觉补丁特征纳入其中以实现视觉与文本的对齐。同时，它还提取粗粒度的视觉特征（例如，整体图像表示）和细粒度的视觉区域（例如，与特定方面相关的区域），并将其转换为相应的文本描述（如面部、美学）。最后，我们利用大型语言模型（LLM）生成的情感原因和印象来增强模型对由语义内容和情感能动-认知共鸣引发的情感线索的理解。 在标准多模态情感分类数据集上的实验结果表明了所提出模型的有效性。与像GPT-4o这样的LLMs相比，该模型表现出更大的灵活性以适应MASC任务。我们在以下链接公开发布了完整实现及数据集：[此URL]
URL
https://arxiv.org/abs/2504.15848
