Abstract
Traditional Multi-Object Tracking (MOT) systems have achieved remarkable precision in localization and association, effectively answering \textit{where} and \textit{who}. However, they often function as autistic observers, capable of tracing geometric paths but blind to the semantic \textit{what} and \textit{why} behind object behaviors. To bridge the gap between geometric perception and cognitive reasoning, we propose \textbf{LLMTrack}, a novel end-to-end framework for Semantic Multi-Object Tracking (SMOT). We adopt a bionic design philosophy that decouples strong localization from deep understanding, utilizing Grounding DINO as the eyes and the LLaVA-OneVision multimodal large model as the brain. We introduce a Spatio-Temporal Fusion Module that aggregates instance-level interaction features and video-level contexts, enabling the Large Language Model (LLM) to comprehend complex trajectories. Furthermore, we design a progressive three-stage training strategy, Visual Alignment, Temporal Fine-tuning, and Semantic Injection via LoRA to efficiently adapt the massive model to the tracking domain. Extensive experiments on the BenSMOT benchmark demonstrate that LLMTrack achieves state-of-the-art performance, significantly outperforming existing methods in instance description, interaction recognition, and video summarization while maintaining robust tracking stability.
Abstract (translated)
传统的多目标跟踪（MOT）系统在定位和关联方面达到了很高的精度，有效地回答了“在哪里”和“谁”的问题。然而，这些系统往往像自闭症观察者一样工作，能够追踪几何路径，但却对物体行为背后的语义上的“是什么”和“为什么”视而不见。为了弥合几何感知与认知推理之间的差距，我们提出了**LLMTrack**，这是一种新的端到端框架，用于语义多目标跟踪（SMOT）。我们的设计哲学借鉴了仿生学原理，将强大的定位功能与深度理解分离，使用Grounding DINO作为“眼睛”以及LLaVA-OneVision多模态大型模型作为“大脑”。我们引入了一个时空融合模块，该模块汇聚实例级交互特征和视频级别上下文，使大规模语言模型能够理解和解析复杂的轨迹。此外，为了有效地将庞大的模型适应到跟踪领域，我们设计了一种分阶段的三步训练策略：视觉对齐、时间微调以及通过LoRA进行语义注入。 在BenSMOT基准测试中的广泛实验表明，LLMTrack达到了最先进的性能，在实例描述、交互识别和视频摘要方面显著优于现有方法，同时保持了强大的跟踪稳定性。
URL
https://arxiv.org/abs/2601.06550
