Abstract
Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality this http URL reproducibility, we release our code at: this https URL
Abstract (translated)
多模态情感计算支撑着诸如情感分析和情绪识别等关键任务。然而,标准评估通常假设文本、音频和视觉模式是同样可用的。在实际应用中,某些模式由于脆弱性或成本原因会系统性地缺失,导致不平衡的丢失率和训练偏差,这些单靠任务级指标无法揭示。 我们推出了MissBench,这是一个针对多模态情感任务的标准基准和框架,涵盖了四种常用的情感和情绪数据集上的共享及不平衡丢失率协议。MissBench还定义了两个诊断指标:模式公平指数(Modality Equity Index, MEI)衡量不同模式在缺失模式配置下贡献的公正性;模式学习指数(Modality Learning Index, MLI)通过比较训练期间各模态特定的梯度范数来量化优化不平衡,这些梯度是跨模态相关模块聚合得到的。 代表性的方法族实验表明,在共享丢失率条件下看似稳健的模型在不平衡条件下面临显著的模式不公平性和优化不平衡。这一发现使MissBench及其MEI和MLI成为测试多模态情感模型在实际不完整模式情况下的压力工具及分析工具,以提高研究工作的可重复性。 为了确保再现性,我们发布了我们的代码:[这里可以填写发布地址]
URL
https://arxiv.org/abs/2603.09874