Paper Reading AI Learner

Exploring Modality-Aware Fusion and Decoupled Temporal Propagation for Multi-Modal Object Tracking

2026-03-10 07:10:05
Shilei Wang, Pujian Lai, Dong Gao, Jifeng Ning, Gong Cheng

Abstract

Most existing multimodal trackers adopt uniform fusion strategies, overlooking the inherent differences between modalities. Moreover, they propagate temporal information through mixed tokens, leading to entangled and less discriminative temporal representations. To address these limitations, we propose MDTrack, a novel framework for modality aware fusion and decoupled temporal propagation in multimodal object tracking. Specifically, for modality aware fusion, we allocate dedicated experts to each modality, including infrared, event, depth, and RGB, to process their respective representations. The gating mechanism within the Mixture of Experts dynamically selects the optimal experts based on the input features, enabling adaptive and modality specific fusion. For decoupled temporal propagation, we introduce two separate State Space Model structures to independently store and update the hidden states of the RGB and X modal streams, effectively capturing their distinct temporal information. To ensure synergy between the two temporal representations, we incorporate a set of cross attention modules between the input features of the two SSMs, facilitating implicit information exchange. The resulting temporally enriched features are then integrated into the backbone through another set of cross attention modules, enhancing MDTrack's ability to leverage temporal information. Extensive experiments demonstrate the effectiveness of our proposed method. Both MDTrack S and MDTrack U achieve state of the art performance across five multimodal tracking benchmarks.

Abstract (translated)

大多数现有的多模态跟踪器采用统一的融合策略,忽视了不同模式之间的固有差异。此外,它们通过混合令牌传播时间信息,导致纠缠且不那么具区分性的时序表示。为了解决这些限制,我们提出了MDTrack,这是一个新颖的框架,用于模态感知融合和解耦的时间传播,在多模态目标跟踪中具有创新性。 具体而言,对于模态感知融合,我们在每个模式(包括红外、事件、深度和RGB)上分配了专门的专家来处理它们各自的表示。Mixture of Experts中的门控机制根据输入特征动态选择最佳的专家,从而实现适应性和特定于模态的融合。 对于解耦的时间传播,我们引入了两个独立的状态空间模型结构(SSMs),分别存储和更新RGB和X模式流的隐藏状态,有效捕捉它们各自独特的时序信息。为了确保这两种时间表示之间的协同作用,我们在两个SSM输入特征之间集成了交叉注意力模块组,促进隐式的信息交换。 然后,通过另一组交叉注意力模块将这些经过时间增强的特征整合到骨干网络中,从而增强了MDTrack利用时间信息的能力。广泛的实验展示了我们所提出方法的有效性。无论是MDTrack S还是MDTrack U,在五个多模态跟踪基准测试上都取得了最先进的性能。

URL

https://arxiv.org/abs/2603.09287

PDF

https://arxiv.org/pdf/2603.09287.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot