Paper Reading AI Learner

MMOT: The First Challenging Benchmark for Drone-based Multispectral Multi-Object Tracking

2025-10-14 14:25:17
Tianhao Li, Tingfa Xu, Ying Wang, Haolin Qin, Xu Lin, Jianan Li

Abstract

Drone-based multi-object tracking is essential yet highly challenging due to small targets, severe occlusions, and cluttered backgrounds. Existing RGB-based tracking algorithms heavily depend on spatial appearance cues such as color and texture, which often degrade in aerial views, compromising reliability. Multispectral imagery, capturing pixel-level spectral reflectance, provides crucial cues that enhance object discriminability under degraded spatial conditions. However, the lack of dedicated multispectral UAV datasets has hindered progress in this domain. To bridge this gap, we introduce MMOT, the first challenging benchmark for drone-based multispectral multi-object tracking. It features three key characteristics: (i) Large Scale - 125 video sequences with over 488.8K annotations across eight categories; (ii) Comprehensive Challenges - covering diverse conditions such as extreme small targets, high-density scenarios, severe occlusions, and complex motion; and (iii) Precise Oriented Annotations - enabling accurate localization and reduced ambiguity under aerial perspectives. To better extract spectral features and leverage oriented annotations, we further present a multispectral and orientation-aware MOT scheme adapting existing methods, featuring: (i) a lightweight Spectral 3D-Stem integrating spectral features while preserving compatibility with RGB pretraining; (ii) an orientation-aware Kalman filter for precise state estimation; and (iii) an end-to-end orientation-adaptive transformer. Extensive experiments across representative trackers consistently show that multispectral input markedly improves tracking performance over RGB baselines, particularly for small and densely packed objects. We believe our work will advance drone-based multispectral multi-object tracking research. Our MMOT, code, and benchmarks are publicly available at this https URL.

Abstract (translated)

基于无人机的多目标跟踪在面对小目标、严重遮挡和复杂背景时,虽然至关重要但也极具挑战性。现有的RGB(红绿蓝)基线追踪算法主要依赖于空间外观线索如颜色和纹理,但在航拍视角下这些线索往往会退化,从而影响可靠性。多光谱影像通过捕捉像素级的光谱反射率提供了关键信息,在空间条件较差的情况下可以增强对象的识别度。然而,缺乏专门针对无人机采集多光谱数据集的发展阻碍了该领域的进步。 为了填补这一空白,我们引入了MMOT,这是第一个用于基于无人机的多光谱多目标跟踪的挑战性基准测试平台。它具有三个关键特性:(i)大规模——包含125个视频序列及超过48.8万条注释;(ii)全面的挑战——涵盖了极端小目标、高密度场景、严重遮挡和复杂运动等多种情况;以及(iii)精确的方向标注——这使得在航拍视角下的精确定位成为可能,减少了不确定性。 为了更好地提取光谱特征并利用定向标注信息,我们进一步提出了一种多光谱感知的MOT方案,在保留与RGB预训练兼容性的前提下进行了现有方法的适应性改进。具体来说: (i) 一个轻量级的Spectral 3D-Stem,能够整合光谱特性; (ii) 一种基于方向的认知卡尔曼滤波器,用于准确的状态估计; (iii) 一种端到端的方向自适应变压器。 在多个代表性追踪器上进行广泛实验后显示,相较于RGB基线方法,多光谱输入显著提升了跟踪性能,尤其是在面对小型及密集目标时。我们相信这项工作将推进基于无人机的多光谱多目标跟踪研究领域的发展。我们的MMOT、代码和基准测试数据集已公开发布在以下网址:[https://this-url.com](https://this-url.com)(实际使用时请替换为正确的URL)。

URL

https://arxiv.org/abs/2510.12565

PDF

https://arxiv.org/pdf/2510.12565.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot