Abstract
Referring Multi-Object Tracking (RMOT) aims to track specific targets based on language descriptions and is vital for interactive AI systems such as robotics and autonomous driving. However, existing RMOT models rely solely on 2D RGB data, making it challenging to accurately detect and associate targets characterized by complex spatial semantics (e.g., ``the person closest to the camera'') and to maintain reliable identities under severe occlusion, due to the absence of explicit 3D spatial information. In this work, we propose a novel task, RGBD Referring Multi-Object Tracking (DRMOT), which explicitly requires models to fuse RGB, Depth (D), and Language (L) modalities to achieve 3D-aware tracking. To advance research on the DRMOT task, we construct a tailored RGBD referring multi-object tracking dataset, named DRSet, designed to evaluate models' spatial-semantic grounding and tracking capabilities. Specifically, DRSet contains RGB images and depth maps from 187 scenes, along with 240 language descriptions, among which 56 descriptions incorporate depth-related information. Furthermore, we propose DRTrack, a MLLM-guided depth-referring tracking framework. DRTrack performs depth-aware target grounding from joint RGB-D-L inputs and enforces robust trajectory association by incorporating depth cues. Extensive experiments on the DRSet dataset demonstrate the effectiveness of our framework.
Abstract (translated)
基于语言描述的多目标跟踪（RMOT）旨在追踪特定的目标，这对于机器人和自动驾驶等互动式AI系统至关重要。然而，现有的RMOT模型仅依赖于2D RGB数据，在处理具有复杂空间语义特征的目标时面临挑战（例如，“离相机最近的人”），并且在严重遮挡情况下难以保持可靠的身份识别，因为缺乏明确的3D空间信息。 为此，我们提出了一项新的任务——RGBD基于语言描述的多目标跟踪（DRMOT），该任务要求模型融合RGB、深度（D）和语言（L）模态来实现具有3D感知能力的目标追踪。为了推进DRMOT的研究，我们构建了一个名为DRSet的定制化RGBD基于语言描述的多对象跟踪数据集，专门用于评估模型的空间语义接地能力和追踪性能。具体来说，DRSet包含来自187个场景中的RGB图像和深度图，以及240条语言描述，其中56条描述包含了与深度相关的信息。 此外，我们提出了DRTrack框架，该框架由多模态语言-视觉大规模预训练模型（MLLM）指导，并利用深度信息进行目标跟踪。DRTrack从联合的RGB-D-L输入中执行具有深度感知的目标定位，并通过整合深度线索来强化轨迹关联，从而实现稳健的追踪性能。 在DRSet数据集上的广泛实验验证了我们框架的有效性。
URL
https://arxiv.org/abs/2602.04692
