Paper Reading AI Learner

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

2026-02-25 15:41:31
Yanqiu Yu, Zhifan Jin, Sijia Chen, Tongfei Chu, En Yu, Liman Liu, Wenbing Tao

Abstract

Referring Multi-Object Tracking has attracted increasing attention due to its human-friendly interactive characteristics, yet it exhibits limitations in low-visibility conditions, such as nighttime, smoke, and other challenging scenarios. To overcome this limitation, we propose a new RGB-Thermal RMOT task, named RT-RMOT, which aims to fuse RGB appearance features with the illumination robustness of the thermal modality to enable all-day referring multi-object tracking. To promote research on RT-RMOT, we construct the first Referring Multi-Object Tracking dataset under RGB-Thermal modality, named RefRT. It contains 388 language descriptions, 1,250 tracked targets, and 166,147 Language-RGB-Thermal (L-RGB-T) triplets. Furthermore, we propose RTrack, a framework built upon a multimodal large language model (MLLM) that integrates RGB, thermal, and textual features. Since the initial framework still leaves room for improvement, we introduce a Group Sequence Policy Optimization (GSPO) strategy to further exploit the model's potential. To alleviate training instability during RL fine-tuning, we introduce a Clipped Advantage Scaling (CAS) strategy to suppress gradient explosion. In addition, we design Structured Output Reward and Comprehensive Detection Reward to balance exploration and exploitation, thereby improving the completeness and accuracy of target perception. Extensive experiments on the RefRT dataset demonstrate the effectiveness of the proposed RTrack framework.

Abstract (translated)

基于参考的多对象跟踪(Referring Multi-Object Tracking,RMOT)由于其人性化的互动特性而备受关注,然而,在低可见度条件下如夜间、烟雾和其他具有挑战性的场景中,该技术表现出明显的局限性。为了克服这一限制,我们提出了一项新的RGB-热成像RMOT任务,命名为RT-RMOT(RGB-Thermal Referring Multi-Object Tracking),旨在融合RGB外观特征与热成像模态的照明鲁棒性,以实现全天候参考多对象跟踪。 为促进RT-RMOT的研究,我们构建了首个基于RGB-热成像模式下的参考多目标跟踪数据集,命名为RefRT。该数据集包含388条语言描述、1250个追踪目标和166,147组语言-RGB-热成像(L-RGB-T)三元组。 此外,我们提出了RTrack框架,它是基于一个多模态大型语言模型(Multimodal Large Language Model,MLLM),集成了RGB、热成像和文本特征。尽管初始的框架仍有改进空间,我们引入了分组序列策略优化(Group Sequence Policy Optimization,GSPO)策略来进一步挖掘该模型的潜力。 为了缓解强化学习微调过程中的训练不稳定性问题,我们提出了一种裁剪优势缩放(Clipped Advantage Scaling,CAS)策略以抑制梯度爆炸。同时,设计结构化输出奖励和综合检测奖励机制来平衡探索与利用的关系,从而提高目标感知的完整性和准确性。 在RefRT数据集上进行的广泛实验表明了所提出的RTrack框架的有效性。

URL

https://arxiv.org/abs/2602.22033

PDF

https://arxiv.org/pdf/2602.22033.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot