Paper Reading AI Learner

Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition

2026-04-07 14:42:22
Tianyi Liu, Yiming Li, Wenqian Wang, Jiaojiao Wang, Chen Cai, Yi Wang, Kim-Hui Yap

Abstract

Robust multimodal visual analytics remains challenging when heterogeneous modalities provide complementary but input-dependent evidence for this http URL multimodal learning methods mainly rely on fixed fusion modules or predefined cross-modal interactions, which are often insufficient to adapt to changing modality reliability and to capture fine-grained action cues. To address this issue, we propose a Mixture-of-Modality-Experts (MoME) framework with a Holistic Token Learning (HTL) strategy. MoME enables adaptive collaboration among modality-specific experts, while HTL improves both intra-expert refinement and inter-expert knowledge transfer through class tokens and spatio-temporal tokens. In this way, our method forms a knowledge-centric multimodal learning framework that improves expert specialization while reducing ambiguity in multimodal this http URL validate the proposed framework on driver action recognition as a representative multimodal understanding taskThe experimental results on the public benchmark show that the proposed MoME framework and the HTL strategy jointly outperform representative single-modal and multimodal baselines. Additional ablation, validation, and visualization results further verify that the proposed HTL strategy improves subtle multimodal understanding and offers better interpretability.

Abstract (translated)

稳健的多模态视觉分析仍面临挑战,因为异构模态提供的互补证据往往依赖于输入。当前多模态学习方法主要依赖固定的融合模块或预定义的跨模态交互,这些方法通常不足以适应变化的模态可靠性,也难以捕捉细粒度动作线索。针对这一问题,我们提出了一个结合整体标记学习策略的模态专家混合框架。该框架实现模态特定专家间的自适应协作,而整体标记学习策略则通过类标记和时空标记来增强专家内部细化与专家间知识迁移。由此,我们的方法构建了一个以知识为中心的多模态学习框架,在提升专家专业性的同时降低了多模态歧义。我们在驾驶员动作识别这一代表性多模态理解任务上验证了所提框架。在公开基准数据集上的实验结果表明,所提出的模态专家混合框架与整体标记学习策略联合优于代表性单模态与多模态基线方法。额外的消融实验、验证与可视化结果进一步证明,所提出的整体标记学习策略能改善细微的多模态理解能力,并提供更好的可解释性。

URL

https://arxiv.org/abs/2604.05947

PDF

https://arxiv.org/pdf/2604.05947.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot