Abstract
Robust multimodal visual analytics remains challenging when heterogeneous modalities provide complementary but input-dependent evidence for this http URL multimodal learning methods mainly rely on fixed fusion modules or predefined cross-modal interactions, which are often insufficient to adapt to changing modality reliability and to capture fine-grained action cues. To address this issue, we propose a Mixture-of-Modality-Experts (MoME) framework with a Holistic Token Learning (HTL) strategy. MoME enables adaptive collaboration among modality-specific experts, while HTL improves both intra-expert refinement and inter-expert knowledge transfer through class tokens and spatio-temporal tokens. In this way, our method forms a knowledge-centric multimodal learning framework that improves expert specialization while reducing ambiguity in multimodal this http URL validate the proposed framework on driver action recognition as a representative multimodal understanding taskThe experimental results on the public benchmark show that the proposed MoME framework and the HTL strategy jointly outperform representative single-modal and multimodal baselines. Additional ablation, validation, and visualization results further verify that the proposed HTL strategy improves subtle multimodal understanding and offers better interpretability.
Abstract (translated)
稳健的多模态视觉分析仍面临挑战,因为异构模态提供的互补证据往往依赖于输入。当前多模态学习方法主要依赖固定的融合模块或预定义的跨模态交互,这些方法通常不足以适应变化的模态可靠性,也难以捕捉细粒度动作线索。针对这一问题,我们提出了一个结合整体标记学习策略的模态专家混合框架。该框架实现模态特定专家间的自适应协作,而整体标记学习策略则通过类标记和时空标记来增强专家内部细化与专家间知识迁移。由此,我们的方法构建了一个以知识为中心的多模态学习框架,在提升专家专业性的同时降低了多模态歧义。我们在驾驶员动作识别这一代表性多模态理解任务上验证了所提框架。在公开基准数据集上的实验结果表明,所提出的模态专家混合框架与整体标记学习策略联合优于代表性单模态与多模态基线方法。额外的消融实验、验证与可视化结果进一步证明,所提出的整体标记学习策略能改善细微的多模态理解能力,并提供更好的可解释性。
URL
https://arxiv.org/abs/2604.05947