Paper Reading AI Learner

MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

2026-03-10 16:36:45
Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le, Cam-Van Thi Nguyen

Abstract

Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality this http URL reproducibility, we release our code at: this https URL

Abstract (translated)

多模态情感计算支撑着诸如情感分析和情绪识别等关键任务。然而,标准评估通常假设文本、音频和视觉模式是同样可用的。在实际应用中,某些模式由于脆弱性或成本原因会系统性地缺失,导致不平衡的丢失率和训练偏差,这些单靠任务级指标无法揭示。 我们推出了MissBench,这是一个针对多模态情感任务的标准基准和框架,涵盖了四种常用的情感和情绪数据集上的共享及不平衡丢失率协议。MissBench还定义了两个诊断指标:模式公平指数(Modality Equity Index, MEI)衡量不同模式在缺失模式配置下贡献的公正性;模式学习指数(Modality Learning Index, MLI)通过比较训练期间各模态特定的梯度范数来量化优化不平衡,这些梯度是跨模态相关模块聚合得到的。 代表性的方法族实验表明,在共享丢失率条件下看似稳健的模型在不平衡条件下面临显著的模式不公平性和优化不平衡。这一发现使MissBench及其MEI和MLI成为测试多模态情感模型在实际不完整模式情况下的压力工具及分析工具,以提高研究工作的可重复性。 为了确保再现性,我们发布了我们的代码:[这里可以填写发布地址]

URL

https://arxiv.org/abs/2603.09874

PDF

https://arxiv.org/pdf/2603.09874.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot