Abstract
Most existing multimodal trackers adopt uniform fusion strategies, overlooking the inherent differences between modalities. Moreover, they propagate temporal information through mixed tokens, leading to entangled and less discriminative temporal representations. To address these limitations, we propose MDTrack, a novel framework for modality aware fusion and decoupled temporal propagation in multimodal object tracking. Specifically, for modality aware fusion, we allocate dedicated experts to each modality, including infrared, event, depth, and RGB, to process their respective representations. The gating mechanism within the Mixture of Experts dynamically selects the optimal experts based on the input features, enabling adaptive and modality specific fusion. For decoupled temporal propagation, we introduce two separate State Space Model structures to independently store and update the hidden states of the RGB and X modal streams, effectively capturing their distinct temporal information. To ensure synergy between the two temporal representations, we incorporate a set of cross attention modules between the input features of the two SSMs, facilitating implicit information exchange. The resulting temporally enriched features are then integrated into the backbone through another set of cross attention modules, enhancing MDTrack's ability to leverage temporal information. Extensive experiments demonstrate the effectiveness of our proposed method. Both MDTrack S and MDTrack U achieve state of the art performance across five multimodal tracking benchmarks.
Abstract (translated)
大多数现有的多模态跟踪器采用统一的融合策略,忽视了不同模式之间的固有差异。此外,它们通过混合令牌传播时间信息,导致纠缠且不那么具区分性的时序表示。为了解决这些限制,我们提出了MDTrack,这是一个新颖的框架,用于模态感知融合和解耦的时间传播,在多模态目标跟踪中具有创新性。 具体而言,对于模态感知融合,我们在每个模式(包括红外、事件、深度和RGB)上分配了专门的专家来处理它们各自的表示。Mixture of Experts中的门控机制根据输入特征动态选择最佳的专家,从而实现适应性和特定于模态的融合。 对于解耦的时间传播,我们引入了两个独立的状态空间模型结构(SSMs),分别存储和更新RGB和X模式流的隐藏状态,有效捕捉它们各自独特的时序信息。为了确保这两种时间表示之间的协同作用,我们在两个SSM输入特征之间集成了交叉注意力模块组,促进隐式的信息交换。 然后,通过另一组交叉注意力模块将这些经过时间增强的特征整合到骨干网络中,从而增强了MDTrack利用时间信息的能力。广泛的实验展示了我们所提出方法的有效性。无论是MDTrack S还是MDTrack U,在五个多模态跟踪基准测试上都取得了最先进的性能。
URL
https://arxiv.org/abs/2603.09287