Abstract
Drone-based multi-object tracking is essential yet highly challenging due to small targets, severe occlusions, and cluttered backgrounds. Existing RGB-based tracking algorithms heavily depend on spatial appearance cues such as color and texture, which often degrade in aerial views, compromising reliability. Multispectral imagery, capturing pixel-level spectral reflectance, provides crucial cues that enhance object discriminability under degraded spatial conditions. However, the lack of dedicated multispectral UAV datasets has hindered progress in this domain. To bridge this gap, we introduce MMOT, the first challenging benchmark for drone-based multispectral multi-object tracking. It features three key characteristics: (i) Large Scale - 125 video sequences with over 488.8K annotations across eight categories; (ii) Comprehensive Challenges - covering diverse conditions such as extreme small targets, high-density scenarios, severe occlusions, and complex motion; and (iii) Precise Oriented Annotations - enabling accurate localization and reduced ambiguity under aerial perspectives. To better extract spectral features and leverage oriented annotations, we further present a multispectral and orientation-aware MOT scheme adapting existing methods, featuring: (i) a lightweight Spectral 3D-Stem integrating spectral features while preserving compatibility with RGB pretraining; (ii) an orientation-aware Kalman filter for precise state estimation; and (iii) an end-to-end orientation-adaptive transformer. Extensive experiments across representative trackers consistently show that multispectral input markedly improves tracking performance over RGB baselines, particularly for small and densely packed objects. We believe our work will advance drone-based multispectral multi-object tracking research. Our MMOT, code, and benchmarks are publicly available at this https URL.
Abstract (translated)
基于无人机的多目标跟踪在面对小目标、严重遮挡和复杂背景时,虽然至关重要但也极具挑战性。现有的RGB(红绿蓝)基线追踪算法主要依赖于空间外观线索如颜色和纹理,但在航拍视角下这些线索往往会退化,从而影响可靠性。多光谱影像通过捕捉像素级的光谱反射率提供了关键信息,在空间条件较差的情况下可以增强对象的识别度。然而,缺乏专门针对无人机采集多光谱数据集的发展阻碍了该领域的进步。 为了填补这一空白,我们引入了MMOT,这是第一个用于基于无人机的多光谱多目标跟踪的挑战性基准测试平台。它具有三个关键特性:(i)大规模——包含125个视频序列及超过48.8万条注释;(ii)全面的挑战——涵盖了极端小目标、高密度场景、严重遮挡和复杂运动等多种情况;以及(iii)精确的方向标注——这使得在航拍视角下的精确定位成为可能,减少了不确定性。 为了更好地提取光谱特征并利用定向标注信息,我们进一步提出了一种多光谱感知的MOT方案,在保留与RGB预训练兼容性的前提下进行了现有方法的适应性改进。具体来说: (i) 一个轻量级的Spectral 3D-Stem,能够整合光谱特性; (ii) 一种基于方向的认知卡尔曼滤波器,用于准确的状态估计; (iii) 一种端到端的方向自适应变压器。 在多个代表性追踪器上进行广泛实验后显示,相较于RGB基线方法,多光谱输入显著提升了跟踪性能,尤其是在面对小型及密集目标时。我们相信这项工作将推进基于无人机的多光谱多目标跟踪研究领域的发展。我们的MMOT、代码和基准测试数据集已公开发布在以下网址:[https://this-url.com](https://this-url.com)(实际使用时请替换为正确的URL)。
URL
https://arxiv.org/abs/2510.12565