Abstract
As multi-object tracking (MOT) tasks continue to evolve toward more general and multi-modal scenarios, the rigid and task-specific architectures of existing MOT methods increasingly hinder their applicability across diverse tasks and limit flexibility in adapting to new tracking formulations. Most approaches rely on fixed output heads and bespoke tracking pipelines, making them difficult to extend to more complex or instruction-driven tasks. To address these limitations, we propose AR-MOT, a novel autoregressive paradigm that formulates MOT as a sequence generation task within a large language model (LLM) framework. This design enables the model to output structured results through flexible sequence construction, without requiring any task-specific heads. To enhance region-level visual perception, we introduce an Object Tokenizer based on a pretrained detector. To mitigate the misalignment between global and regional features, we propose a Region-Aware Alignment (RAA) module, and to support long-term tracking, we design a Temporal Memory Fusion (TMF) module that caches historical object tokens. AR-MOT offers strong potential for extensibility, as new modalities or instructions can be integrated by simply modifying the output sequence format without altering the model architecture. Extensive experiments on MOT17 and DanceTrack validate the feasibility of our approach, achieving performance comparable to state-of-the-art methods while laying the foundation for more general and flexible MOT systems.
Abstract (translated)
随着多目标跟踪（MOT）任务向更加通用和多模态场景发展，现有MOT方法的刚性架构及其针对特定任务的设计越来越难以适应多样化的任务需求，并限制了其灵活性以应对新的追踪形式。大多数方法依赖于固定输出头和专用追踪管道，这使得它们很难扩展到更复杂或指令驱动的任务中。为了解决这些局限性，我们提出了AR-MOT，这是一种新型的自回归范式，它将MOT作为大型语言模型（LLM）框架中的序列生成任务来定义。这种设计使模型能够通过灵活的序列构建输出结构化结果，并且无需任何特定于任务的头部即可实现这一点。 为了增强区域级别的视觉感知能力，我们引入了一个基于预训练检测器的对象标记器(Object Tokenizer)。为了解决全局特征与局部特征之间的不匹配问题，我们提出了一个区域感知对齐(RAA)模块；同时，为了支持长期跟踪，我们设计了一个时间记忆融合(TMF)模块，该模块可以缓存历史对象标记。 AR-MOT展示了很强的可扩展性潜力，因为新的模态或指令可以通过简单地修改输出序列格式而不改变模型架构来集成。在MOT17和DanceTrack上的广泛实验验证了我们方法的有效性，实现了与最新技术相媲美的性能，并为更通用、灵活的MOT系统奠定了基础。
URL
https://arxiv.org/abs/2601.01925
