Abstract
Multimodal Sentiment Analysis (MSA) integrates language, visual, and acoustic modalities to infer human sentiment. Most existing methods either focus on globally shared representations or modality-specific features, while overlooking signals that are shared only by certain modality pairs. This limits the expressiveness and discriminative power of multimodal representations. To address this limitation, we propose a Tri-Subspace Disentanglement (TSD) framework that explicitly factorizes features into three complementary subspaces: a common subspace capturing global consistency, submodally-shared subspaces modeling pairwise cross-modal synergies, and private subspaces preserving modality-specific cues. To keep these subspaces pure and independent, we introduce a decoupling supervisor together with structured regularization losses. We further design a Subspace-Aware Cross-Attention (SACA) fusion module that adaptively models and integrates information from the three subspaces to obtain richer and more robust representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that TSD achieves state-of-the-art performance across all key metrics, reaching 0.691 MAE on CMU-MOSI and 54.9% ACC-7 on CMU-MOSEI, and also transfers well to multimodal intent recognition tasks. Ablation studies confirm that tri-subspace disentanglement and SACA jointly enhance the modeling of multi-granular cross-modal sentiment cues.
Abstract (translated)
多模态情感分析(MSA)整合语言、视觉和听觉模式来推断人类情绪。现有大多数方法要么侧重于全局共享表示,要么关注特定模态的特征,而忽略了仅由某些模态对共有的信号。这限制了多模态表示的表达能力和判别能力。为了解决这一局限性,我们提出了一种三子空间解耦(TSD)框架,该框架明确地将特征分解成三个互补的子空间:一个捕捉全局一致性的公共子空间、两个跨模式协同作用的子模态共享子空间以及保留特定模态线索的私有子空间。为了保持这些子空间的纯净和独立性,我们引入了一个解耦监督器,并结合结构化正则化损失。此外,我们还设计了一种感知子空间的交叉注意力(SACA)融合模块,该模块能够自适应地从三个子空间中建模和集成信息,以获得更丰富、更稳健的表示。 在CMU-MOSI和CMU-MOSEI数据集上的实验表明,TSD框架实现了最先进的性能,在所有关键指标上均表现出色:在CMU-MOSI上达到0.691 MAE(平均绝对误差);在CMU-MOSEI上实现54.9%的ACC-7准确率,并且该模型还能很好地转移到多模态意图识别任务中。消融研究表明,三子空间解耦和SACA模块共同提高了对跨模式情感线索的多层次建模能力。 总结来说,TSD框架通过引入更精细的特征表示和融合机制,在多模态情感分析任务上取得了显著的进步,并展示了良好的泛化能力和应用潜力。
URL
https://arxiv.org/abs/2602.19585