Abstract
Multi-Object Tracking (MOT) has traditionally focused on a few specific categories, restricting its applicability to real-world scenarios involving diverse objects. Open-Vocabulary Multi-Object Tracking (OVMOT) addresses this by enabling tracking of arbitrary categories, including novel objects unseen during training. However, current progress is constrained by two challenges: the lack of continuously annotated video data for training, and the lack of a customized OVMOT framework to synergistically handle detection and association. We address the data bottleneck by constructing C-TAO, the first continuously annotated training set for OVMOT, which increases annotation density by 26x over the original TAO and captures smooth motion dynamics and intermediate object states. For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules: (1) Multi-Cue Adaptive Fusion (MCF) dynamically balances appearance, motion, and semantic cues for association feature learning; (2) Multi-Granularity Hierarchical Aggregation (MGA) exploits hierarchical spatial relationships in dense detections, where visible child nodes (e.g., object parts) assist occluded parent objects (e.g., whole body) for association feature enhancement; (3) Temporal Confidence Propagation (TCP) recovers flickering detections through high-confidence tracked objects boosting low-confidence candidates across frames, stabilizing trajectories. Extensive experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods, and show strong zero-shot generalization on BDD100K. The code and dataset will be publicly available.
Abstract (translated)
多目标跟踪（MOT）传统上聚焦于少数特定类别，限制了其在涉及多样化物体的真实场景中的适用性。开放词汇多目标跟踪（OVMOT）通过实现对任意类别的跟踪（包括训练期间未见的新物体）解决了这一问题。然而，当前进展受限于两大挑战：缺乏用于训练的连续标注视频数据，以及缺乏定制化的OVMOT框架来协同处理检测与关联。我们通过构建C-TAO（首个面向OVMOT的连续标注训练集）解决了数据瓶颈，其标注密度较原始TAO数据集提升26倍，并捕捉了平滑的运动动态与中间物体状态。针对框架瓶颈，我们提出了协同框架COVTrack++，通过三个模块实现检测与关联之间的双向互惠机制：（1）多线索自适应融合（MCF）动态平衡外观、运动与语义线索以进行关联特征学习；（2）多粒度分层聚合（MGA）利用密集检测中的分层空间关系，其中可见子节点（如物体部件）辅助被遮挡的父对象（如完整身体）以增强关联特征；（3）时序置信度传播（TCP）通过高置信度跟踪物体跨帧提升低置信度候选检测，恢复闪烁检测结果并稳定轨迹。在TAO数据集上的大量实验表明该方法达到最先进性能，新类别TETA指标在验证集与测试集分别达到35.4%与30.5%，较先前方法将新类别AssocA提升4.8%、LocA提升5.8%，并在BDD100K上展现出强零样本泛化能力。代码与数据集将公开可用。
URL
https://arxiv.org/abs/2603.24016
