Object_Tracking
-
Quadratic Kalman Filter for Elliptical Extended Object Tracking based on Decoupling State Components
2025-12-16 14:15:40Simon Steuernagel, Marcus BaumAbstract
Extended object tracking involves estimating both the physical extent and kinematic parameters of a target object, where typically multiple measurements are observed per time step. In this article, we propose a deterministic closed-form elliptical extended object tracker, based on decoupling of the kinematics, orientation, and axis lengths. By disregarding potential correlations between these state components, fewer approximations are required for the individual estimators than for an overall joint solution. The resulting algorithm outperforms existing algorithms, reaching the accuracy of sampling-based procedures. Additionally, a batch-based variant is introduced, yielding highly efficient computation while outperforming all comparable state-of-the-art algorithms. This is validated both by a simulation study using common models from literature, as well as an extensive quantitative evaluation on real automotive radar data.
Abstract (translated)
扩展目标跟踪涉及估算目标对象的物理范围和运动学参数，通常在每个时间步骤中会观察到多个测量值。本文提出了一种基于确定性闭式椭圆扩展目标追踪器，该方法通过分离运动学、姿态和轴长来实现。通过忽略这些状态组件之间的潜在相关性，对于单独的估计器比总体联合解决方案所需的近似更少。由此产生的算法优于现有的算法，并且达到了采样法程序的精度水平。此外，还引入了一种基于批处理的方法，该方法计算效率极高，并在所有同类最先进的算法中表现最佳。这一结论既通过使用文献中的常见模型进行仿真研究得到验证，也通过大量的定量评估实际汽车雷达数据得到了证实。
URL
https://arxiv.org/abs/2512.14426
PDF
-
Recurrent Video Masked Autoencoders
2025-12-15 18:59:48Daniel Zoran, Nikhil Parthasarathy, Yi Yang, Drew A Hudson, Joao Carreira, Andrew ZissermanAbstract
We present Recurrent Video Masked-Autoencoders (RVM): a novel video representation learning approach that uses a transformer-based recurrent neural network to aggregate dense image features over time, effectively capturing the spatio-temporal structure of natural video data. RVM learns via an asymmetric masked prediction task requiring only a standard pixel reconstruction objective. This design yields a highly efficient ``generalist'' encoder: RVM achieves competitive performance with state-of-the-art video models (e.g. VideoMAE, V-JEPA) on video-level tasks like action recognition and point/object tracking, while also performing favorably against image models (e.g. DINOv2) on tasks that test geometric and dense spatial understanding. Notably, RVM achieves strong performance in the small-model regime without requiring knowledge distillation, exhibiting up to 30x greater parameter efficiency than competing video masked autoencoders. Moreover, we demonstrate that RVM's recurrent nature allows for stable feature propagation over long temporal horizons with linear computational cost, overcoming some of the limitations of standard spatio-temporal attention-based architectures. Finally, we use qualitative visualizations to highlight that RVM learns rich representations of scene semantics, structure, and motion.
Abstract (translated)
我们提出了递归视频掩码自编码器（RVM）：一种新颖的视频表示学习方法，它使用基于变压器的循环神经网络来在时间维度上聚合密集图像特征，从而有效地捕捉自然视频数据中的空间-时间结构。RVM 通过一个非对称的掩码预测任务进行学习，该任务只需要标准像素重建目标即可完成。这种设计产生了一个高效的“通才”编码器：RVM 在诸如动作识别和点/对象跟踪等视频级别的任务上表现出与最先进的视频模型（如 VideoMAE、V-JEPA）相当的性能，并且在测试几何学和密集空间理解的任务中，其表现也优于图像模型（例如 DINOv2）。值得注意的是，即使不使用知识蒸馏，在小规模模型环境下 RVM 也能取得优异的成绩，比竞争中的视频掩码自编码器参数效率高出多达30倍。此外，我们证明了由于 RVM 的递归特性，它能够在长时序范围内以线性计算成本稳定地传播特征，克服了一些基于空间-时间注意的架构的限制。最后，我们通过定性的可视化展示了 RVM 学习到了丰富的场景语义、结构和运动表示。
URL
https://arxiv.org/abs/2512.13684
PDF
-
LeafTrackNet: A Deep Learning Framework for Robust Leaf Tracking in Top-Down Plant Phenotyping
2025-12-15 09:43:07Shanghua Liu, Majharulislam Babor, Christoph Verduyn, Breght Vandenberghe, Bruno Betoni Parodi, Cornelia Weltzien, Marina M. -C. H\"ohneAbstract
High resolution phenotyping at the level of individual leaves offers fine-grained insights into plant development and stress responses. However, the full potential of accurate leaf tracking over time remains largely unexplored due to the absence of robust tracking methods-particularly for structurally complex crops such as canola. Existing plant-specific tracking methods are typically limited to small-scale species or rely on constrained imaging conditions. In contrast, generic multi-object tracking (MOT) methods are not designed for dynamic biological scenes. Progress in the development of accurate leaf tracking models has also been hindered by a lack of large-scale datasets captured under realistic conditions. In this work, we introduce CanolaTrack, a new benchmark dataset comprising 5,704 RGB images with 31,840 annotated leaf instances spanning the early growth stages of 184 canola plants. To enable accurate leaf tracking over time, we introduce LeafTrackNet, an efficient framework that combines a YOLOv10-based leaf detector with a MobileNetV3-based embedding network. During inference, leaf identities are maintained over time through an embedding-based memory association strategy. LeafTrackNet outperforms both plant-specific trackers and state-of-the-art MOT baselines, achieving a 9% HOTA improvement on CanolaTrack. With our work we provide a new standard for leaf-level tracking under realistic conditions and we provide CanolaTrack - the largest dataset for leaf tracking in agriculture crops, which will contribute to future research in plant phenotyping. Our code and dataset are publicly available at this https URL.
Abstract (translated)
高分辨率的单叶表型分析为植物发育和压力响应提供了精细的见解。然而，由于缺乏稳健的追踪方法（特别是对于结构复杂的作物如油菜），随着时间推移准确地进行叶片跟踪的全部潜力尚未被充分发掘。现有的特定于植物的跟踪方法通常仅限于小型物种或依赖于受限的成像条件。相比之下，通用多目标跟踪（MOT）方法并未针对动态生物场景设计。由于缺乏在现实条件下捕获的大规模数据集，准确叶片追踪模型的发展也受到了阻碍。 在这项工作中，我们介绍了CanolaTrack，这是一个新的基准数据集，包含5,704张RGB图像和31,840个注释的叶子实例，涵盖了184株油菜植物的早期生长阶段。为了实现在时间上的准确叶片跟踪，我们引入了LeafTrackNet，这是一种高效的框架，结合了一种基于YOLOv10的叶检测器和一种基于MobileNetV3的嵌入式网络。在推断过程中，通过一种基于嵌入式的记忆关联策略，在一段时间内保持叶子的身份信息。LeafTrackNet超越了特定于植物的跟踪器和最先进的MOT基线方法，在CanolaTrack上实现了9%的HOTA改进。我们提供的这项工作为现实条件下叶片级追踪提供了一个新的标准，并提供了CanolaTrack——农业作物中最大的叶片跟踪数据集，这将有助于未来在植物表型研究中的发展。 我们的代码和数据集可以在[此链接]（https://给出URL）公开获取。
URL
https://arxiv.org/abs/2512.13130
PDF
-
Light Field Based 6DoF Tracking of Previously Unobserved Objects
2025-12-15 06:04:49Nikolai Goncharov, James L. Gray, Donald G. DansereauAbstract
Object tracking is an important step in robotics and reautonomous driving pipelines, which has to generalize to previously unseen and complex objects. Existing high-performing methods often rely on pre-captured object views to build explicit reference models, which restricts them to a fixed set of known objects. However, such reference models can struggle with visually complex appearance, reducing the quality of tracking. In this work, we introduce an object tracking method based on light field images that does not depend on a pre-trained model, while being robust to complex visual behavior, such as reflections. We extract semantic and geometric features from light field inputs using vision foundation models and convert them into view-dependent Gaussian splats. These splats serve as a unified object representation, supporting differentiable rendering and pose optimization. We further introduce a light field object tracking dataset containing challenging reflective objects with precise ground truth poses. Experiments demonstrate that our method is competitive with state-of-the-art model-based trackers in these difficult cases, paving the way toward universal object tracking in robotic systems. Code/data available at this https URL.
Abstract (translated)
对象跟踪是机器人和自动驾驶流水线中的一个重要步骤，需要对以前未见过的复杂物体进行泛化。现有的高性能方法通常依赖于预捕获的对象视图来构建显式的参考模型，这限制了它们只能处理一组已知的对象。然而，这样的参考模型在面对视觉上复杂的外观时会遇到困难，从而降低了跟踪质量。在这项工作中，我们介绍了一种基于光场图像的对象跟踪方法，该方法无需依赖预训练的模型，并且对诸如反射等复杂视觉行为具有鲁棒性。我们利用视觉基础模型从光场输入中提取语义和几何特征，并将其转换为视图相关的高斯点（Gaussian splats）。这些点作为统一的对象表示形式，支持可微渲染和姿态优化。此外，我们还引入了一个包含挑战性的反射物体的光场对象跟踪数据集，并提供了精确的姿态地面实况。实验表明，在这些困难的情况下，我们的方法与最先进的模型基于的方法具有竞争力，为机器人系统中的通用对象跟踪铺平了道路。代码和数据可在该链接获取。
URL
https://arxiv.org/abs/2512.13007
PDF
-
MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation
2025-12-11 18:59:44Henghui Ding, Chang Liu, Shuting He, Kaining Ying, Xudong Jiang, Chen Change Loy, Yu-Gang JiangAbstract
This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects' motions. Existing referring video segmentation datasets often focus on salient objects and use language expressions rich in static attributes, potentially allowing the target object to be identified in a single frame. Such datasets underemphasize the role of motion in both videos and languages. To explore the feasibility of using motion expressions and motion reasoning clues for pixel-level video understanding, we introduce MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio, covering 8,171 objects in 2,006 videos of complex scenarios. We benchmark 15 existing methods across 4 tasks supported by MeViS, including 6 referring video object segmentation (RVOS) methods, 3 audio-guided video object segmentation (AVOS) methods, 2 referring multi-object tracking (RMOT) methods, and 4 video captioning methods for the newly introduced referring motion expression generation (RMEG) task. The results demonstrate weaknesses and limitations of existing methods in addressing motion expression-guided video understanding. We further analyze the challenges and propose an approach LMPM++ for RVOS/AVOS/RMOT that achieves new state-of-the-art results. Our dataset provides a platform that facilitates the development of motion expression-guided video understanding algorithms in complex video scenes. The proposed MeViS dataset and the method's source code are publicly available at this https URL
Abstract (translated)
这篇论文提出了一种大规模的多模态数据集，用于根据对象运动的语言描述对视频进行参考性动作表达分割。该数据集着重于依据语言描述中的物体运动来分割和追踪视频中的目标对象。现有的参考视频分割数据集往往关注显眼的对象，并使用富含静态属性的语言表达方式，这使得在单帧图像中识别出目标对象成为可能。然而，这些数据集低估了动作在视频及语言理解中的作用。 为了探索利用动作表达和运动推理线索进行像素级视频理解的可能性，我们引入了一个名为MeViS的数据集，它包含了33,072个人工注释的动作表达文本与音频，涵盖了复杂场景中2,006个视频里的8,171个物体。我们在MeViS支持的4项任务上基准测试了15种现有方法，包括6种参考视频对象分割（RVOS）方法、3种基于语音引导的视频对象分割（AVOS）方法、2种参考多目标跟踪（RMOT）方法以及针对新提出的参考动作表达生成（RMEG）任务的4种视频描述方法。测试结果揭示了现有方法在处理由动作表达指导的视频理解时所存在的弱点和局限性。 我们进一步分析了面临的挑战，并提出了一种名为LMPM++的新方法，该方法适用于RVOS、AVOS及RMOT，在这些领域实现了新的最佳性能。我们的数据集为复杂场景下基于动作表达的视频理解算法的发展提供了一个平台。MeViS数据集和所提方法的源代码已在以下网址公开发布：[URL]
URL
https://arxiv.org/abs/2512.10945
PDF
-
Benchmarking SAM2-based Trackers on FMOX
2025-12-10 13:21:09Senem Aktas, Charles Markham, John McDonald, Rozenn DahyotAbstract
Several object tracking pipelines extending Segment Anything Model 2 (SAM2) have been proposed in the past year, where the approach is to follow and segment the object from a single exemplar template provided by the user on a initialization frame. We propose to benchmark these high performing trackers (SAM2, EfficientTAM, DAM4SAM and SAMURAI) on datasets containing fast moving objects (FMO) specifically designed to be challenging for tracking approaches. The goal is to understand better current limitations in state-of-the-art trackers by providing more detailed insights on the behavior of these trackers. We show that overall the trackers DAM4SAM and SAMURAI perform well on more challenging sequences.
Abstract (translated)
在过去的一年中，提出了几种基于Segment Anything Model 2（SAM2）的多目标跟踪流水线方法，这些方法旨在通过用户在初始化帧上提供的单个示例模板来追踪和分割对象。我们提议对包含快速移动物体（FMO）的数据集进行基准测试，该数据集专门设计用于挑战现有的跟踪方法。我们的目标是通过提供更详细的关于这些跟踪器行为的见解，更好地理解当前最先进的跟踪器的局限性。结果显示，在更具挑战性的序列上，DAM4SAM和SAMURAI这两款追踪器整体表现较好。
URL
https://arxiv.org/abs/2512.09633
PDF
-
Efficient Feature Compression for Machines with Global Statistics Preservation
2025-12-10 01:51:34Md Eimran Hossain Eimon, Hyomin Choi, Fabien Racap\'e, Mateen Ulhaq, Velibor Adzic, Hari Kalva, Borko FurhtAbstract
The split-inference paradigm divides an artificial intelligence (AI) model into two parts. This necessitates the transfer of intermediate feature data between the two halves. Here, effective compression of the feature data becomes vital. In this paper, we employ Z-score normalization to efficiently recover the compressed feature data at the decoder side. To examine the efficacy of our method, the proposed method is integrated into the latest Feature Coding for Machines (FCM) codec standard under development by the Moving Picture Experts Group (MPEG). Our method supersedes the existing scaling method used by the current standard under development. It both reduces the overhead bits and improves the end-task accuracy. To further reduce the overhead in certain circumstances, we also propose a simplified method. Experiments show that using our proposed method shows 17.09% reduction in bitrate on average across different tasks and up to 65.69% for object tracking without sacrificing the task accuracy.
Abstract (translated)
分而推理的范式将人工智能（AI）模型分为两个部分。这需要在两半之间传输中间特征数据。在此过程中，有效压缩这些特征数据变得至关重要。本文中，我们采用Z分数标准化方法，在解码器端高效地恢复压缩后的特征数据。为了检验该方法的有效性，我们将提出的方法集成到了由国际电信联盟视频编码专家组（MPEG）开发的最新《机器特征编码》（Feature Coding for Machines, FCM）编解码标准中。我们的方法超越了当前正在开发的标准使用的现有缩放方法，在减少开销比特的同时提高了终端任务准确性。为了进一步在某些情况下降低开销，我们还提出了一种简化的方法。实验表明，使用我们提出的方法在不同任务中的平均比特率减少了17.09%，对于物体跟踪任务甚至最多可减少65.69%，而不会牺牲任务的精度。
URL
https://arxiv.org/abs/2512.09235
PDF
-
GorillaWatch: An Automated System for In-the-Wild Gorilla Re-Identification and Population Monitoring
2025-12-08 17:58:20Maximilian Schall, Felix Leonard Kn\"ofel, Noah Elias K\"onig, Jan Jonas Kubeler, Maximilian von Klinski, Joan Wilhelm Linnemann, Xiaoshi Liu, Iven Jelle Schlegelmilch, Ole Woyciniuk, Alexandra Schild, Dante Wasmuht, Magdalena Bermejo Espinet, German Illera Basas, Gerard de MeloAbstract
Monitoring critically endangered western lowland gorillas is currently hampered by the immense manual effort required to re-identify individuals from vast archives of camera trap footage. The primary obstacle to automating this process has been the lack of large-scale, "in-the-wild" video datasets suitable for training robust deep learning models. To address this gap, we introduce a comprehensive benchmark with three novel datasets: Gorilla-SPAC-Wild, the largest video dataset for wild primate re-identification to date; Gorilla-Berlin-Zoo, for assessing cross-domain re-identification generalization; and Gorilla-SPAC-MoT, for evaluating multi-object tracking in camera trap footage. Building on these datasets, we present GorillaWatch, an end-to-end pipeline integrating detection, tracking, and re-identification. To exploit temporal information, we introduce a multi-frame self-supervised pretraining strategy that leverages consistency in tracklets to learn domain-specific features without manual labels. To ensure scientific validity, a differentiable adaptation of AttnLRP verifies that our model relies on discriminative biometric traits rather than background correlations. Extensive benchmarking subsequently demonstrates that aggregating features from large-scale image backbones outperforms specialized video architectures. Finally, we address unsupervised population counting by integrating spatiotemporal constraints into standard clustering to mitigate over-segmentation. We publicly release all code and datasets to facilitate scalable, non-invasive monitoring of endangered species
Abstract (translated)
监测极度濒危的西部低地大猩猩目前受到大量手工劳动的影响，这些工作是为了从大量的相机陷阱视频资料中重新识别个体。自动处理这一过程的主要障碍是缺乏适合训练稳健深度学习模型的大规模、真实环境下的视频数据集。为了填补这一空白，我们引入了一个全面的基准，并提供了三个新的数据集：Gorilla-SPAC-Wild 是迄今为止最大的用于野外灵长类动物再识别的视频数据集；Gorilla-Berlin-Zoo 用于评估跨域再识别泛化能力；以及 Gorilla-SPAC-MoT，用于评估相机陷阱视频中的多目标跟踪性能。基于这些数据集，我们提出了 GorillaWatch，一个端到端的工作流程，整合了检测、追踪和再识别功能。 为了利用时间信息，我们引入了一种跨多个帧的自监督预训练策略，该策略利用轨迹的一致性来学习领域特定特征，而无需手动标签。为确保科学有效性，我们采用可微分的 AttnLRP 来验证我们的模型是否依赖于有鉴别性的生物标志特征而非背景相关因素。 广泛的基准测试随后证明，从大规模图像骨干网络中聚合特征优于专门针对视频架构的方法。最后，为了处理无监督的人口计数问题，我们将时空约束整合到标准聚类中以减少过度分割现象。 我们公开发布所有代码和数据集，以便为濒危物种提供可扩展、非侵入性的监测手段。
URL
https://arxiv.org/abs/2512.07776
PDF
-
NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks
2025-12-06 02:51:19Fangzhou Lin, Yuping Wang, Yuliang Guo, Zixun Huang, Xinyu Huang, Haichong Zhang, Kazunori Yamada, Zhengzhong Tu, Liu Ren, Ziming ZhangAbstract
Partially Supervised Multi-Task Learning (PS-MTL) aims to leverage knowledge across tasks when annotations are incomplete. Existing approaches, however, have largely focused on the simpler setting of homogeneous, dense prediction tasks, leaving the more realistic challenge of learning from structurally diverse tasks unexplored. To this end, we introduce NexusFlow, a novel, lightweight, and plug-and-play framework effective in both settings. NexusFlow introduces a set of surrogate networks with invertible coupling layers to align the latent feature distributions of tasks, creating a unified representation that enables effective knowledge transfer. The coupling layers are bijective, preserving information while mapping features into a shared canonical space. This invertibility avoids representational collapse and enables alignment across structurally different tasks without reducing expressive capacity. We first evaluate NexusFlow on the core challenge of domain-partitioned autonomous driving, where dense map reconstruction and sparse multi-object tracking are supervised in different geographic regions, creating both structural disparity and a strong domain gap. NexusFlow sets a new state-of-the-art result on nuScenes, outperforming strong partially supervised baselines. To demonstrate generality, we further test NexusFlow on NYUv2 using three homogeneous dense prediction tasks, segmentation, depth, and surface normals, as a representative N-task PS-MTL scenario. NexusFlow yields consistent gains across all tasks, confirming its broad applicability.
Abstract (translated)
部分监督多任务学习（PS-MTL）旨在当标注数据不完整时，在不同任务间共享知识。然而，现有的方法主要集中在同质且密集预测任务的简化场景上，而忽略了更为现实的学习挑战：从结构多样化的任务中进行学习。为此，我们引入了NexusFlow框架，这是一个新颖、轻量级且即插即用的框架，适用于上述两种设置。NexusFlow通过引入一组带有可逆耦合层的替代网络来调整各任务的潜在特征分布，从而创建统一表示并实现有效的知识迁移。这些耦合层是双射映射，在将特征映射到共享的标准空间时保留信息。这种可逆性避免了表示塌陷，并允许跨结构不同的任务对齐而不减少表达能力。 我们首先在领域分割的自动驾驶核心挑战上评估NexusFlow，其中密集地图重建和稀疏多对象跟踪分别在一个地理区域中进行监督学习，导致结构差异明显且存在较强的领域差距。NexusFlow在nuScenes数据集上取得了新的最佳结果，超过了强大的部分监督基准模型。 为了展示其通用性，我们进一步使用NYUv2数据集测试了NexusFlow，在该数据集中采用三个同质密集预测任务：分割、深度估计和表面法线作为具有代表性的N任务PS-MTL场景。在所有这些任务中，NexusFlow都取得了持续的改进，证实其广泛的适用性。 总结来说，NexusFlow通过引入可逆耦合层来解决结构化多样化任务中的知识迁移问题，并且已经在多种设置下证明了其有效性与广泛的应用潜力。
URL
https://arxiv.org/abs/2512.06251
PDF
-
Two-Stage Camera Calibration Method for Multi-Camera Systems Using Scene Geometry
Abstract
Calibration of multi-camera systems is a key task for accurate object tracking. However, it remains a challenging problem in real-world conditions, where traditional methods are not applicable due to the lack of accurate floor plans, physical access to place calibration patterns, or synchronized video streams. This paper presents a novel two-stage calibration method that overcomes these limitations. In the first stage, partial calibration of individual cameras is performed based on an operator's annotation of natural geometric primitives (parallel, perpendicular, and vertical lines, or line segments of equal length). This allows estimating key parameters (roll, pitch, focal length) and projecting the camera's Effective Field of View (EFOV) onto the horizontal plane in a base 3D coordinate system. In the second stage, precise system calibration is achieved through interactive manipulation of the projected EFOV polygons. The operator adjusts their position, scale, and rotation to align them with the floor plan or, in its absence, using virtual calibration elements projected onto all cameras in the system. This determines the remaining extrinsic parameters (camera position and yaw). Calibration requires only a static image from each camera, eliminating the need for physical access or synchronized video. The method is implemented as a practical web service. Comparative analysis and demonstration videos confirm the method's applicability, accuracy, and flexibility, enabling the deployment of precise multi-camera tracking systems in scenarios previously considered infeasible.
Abstract (translated)
多摄像头系统的校准是实现精确对象跟踪的关键任务。然而，在实际环境中，由于缺乏准确的平面图、放置校准图案的物理访问权限或同步视频流等原因，传统方法难以适用，使得该问题仍然具有挑战性。本文提出了一种新颖的两阶段校准方法，以克服这些限制。 在第一阶段，根据操作员对自然几何特征（平行线、垂直线和等长线段）的手动标记进行单个摄像头的部分校准。这允许估算关键参数（滚转角、俯仰角、焦距），并将在基础三维坐标系中的有效视野（EFOV）投影到水平面上。 在第二阶段，通过交互式调整投影的EFOV多边形的位置、比例和旋转来实现系统的精确校准。操作员根据平面图调整这些参数以使多边形对齐；若没有平面图，则使用虚拟校准元素投射至系统中的所有摄像头进行调整。这确定了剩余的外参（摄像机位置和偏航角）。该方法仅需每个摄像头的一张静态图像即可完成校准，无需物理访问或同步视频流。 本文的方法已实现为实用的网页服务，并通过比较分析和演示视频证明其适用性、准确性和灵活性，使在以前被认为不可行的情况下部署精确多摄像头跟踪系统成为可能。
URL
https://arxiv.org/abs/2512.05171
PDF
-
TrackNetV5: Residual-Driven Spatio-Temporal Refinement and Motion Direction Decoupling for Fast Object Tracking
2025-12-02 14:04:30Tang Haonan, Chen Yanjun, Jiang LezhiAbstract
The TrackNet series has established a strong baseline for fast-moving small object tracking in sports. However, existing iterations face significant limitations: V1-V3 struggle with occlusions due to a reliance on purely visual cues, while TrackNetV4, despite introducing motion inputs, suffers from directional ambiguity as its absolute difference method discards motion polarity. To overcome these bottlenecks, we propose TrackNetV5, a robust architecture integrating two novel mechanisms. First, to recover lost directional priors, we introduce the Motion Direction Decoupling (MDD) module. Unlike V4, MDD decomposes temporal dynamics into signed polarity fields, explicitly encoding both movement occurrence and trajectory direction. Second, we propose the Residual-Driven Spatio-Temporal Refinement (R-STR) head. Operating on a coarse-to-fine paradigm, this Transformer-based module leverages factorized spatio-temporal contexts to estimate a corrective residual, effectively recovering occluded targets. Extensive experiments on the TrackNetV2 dataset demonstrate that TrackNetV5 achieves a new state-of-the-art F1-score of 0.9859 and an accuracy of 0.9733, significantly outperforming previous versions. Notably, this performance leap is achieved with a marginal 3.7% increase in FLOPs compared to V4, maintaining real-time inference capabilities while delivering superior tracking precision.
Abstract (translated)
TrackNet系列为体育赛事中快速移动的小物体跟踪建立了强大的基准。然而，现有的版本面临一些显著的限制：V1-V3由于完全依赖于视觉线索，在处理遮挡时表现不佳；而TrackNetV4虽然引入了运动输入，但由于其绝对差分方法放弃了运动极性信息，导致方向模糊问题依然存在。为克服这些瓶颈，我们提出了TrackNetV5这一稳健架构，集成了两种新颖的机制。 首先，为了恢复丢失的方向先验知识，我们介绍了运动方向解耦（MDD）模块。不同于V4版本，MDD将时间动力学分解成带有符号极性场的形式，明确编码了移动的发生和轨迹方向。其次，我们提出了残差驱动的空间-时间细化（R-STR）头部。这一基于Transformer的模块采用从粗到细的工作模式，利用因子化的时间-空间上下文来估计校正残差，从而有效恢复被遮挡的目标。 在TrackNetV2数据集上的广泛实验表明，TrackNetV5达到了0.9859的新F1评分和0.9733的精度，显著优于之前的版本。值得注意的是，这种性能飞跃仅伴随着与V4相比微小的3.7%浮点运算（FLOPs）增加，在保持实时推理能力的同时提供了更精确的跟踪效果。
URL
https://arxiv.org/abs/2512.02789
PDF
-
From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking
2025-12-02 04:04:39Yuqing Shao, Yuchen Yang, Rui Yu, Weilong Li, Xu Guo, Huaicheng Yan, Wei Wang, Xiao SunAbstract
End-to-end multi-object tracking (MOT) methods have recently achieved remarkable progress by unifying detection and association within a single framework. Despite their strong detection performance, these methods suffer from relatively low association accuracy. Through detailed analysis, we observe that object embeddings produced by the shared DETR architecture display excessively high inter-object similarity, as it emphasizes only category-level discrimination within single frames. In contrast, tracking requires instance-level distinction across frames with spatial and temporal continuity, for which current end-to-end approaches insufficiently optimize object embeddings. To address this, we introduce FDTA (From Detection to Association), an explicit feature refinement framework that enhances object discriminativeness across three complementary perspectives. Specifically, we introduce a Spatial Adapter (SA) to integrate depth-aware cues for spatial continuity, a Temporal Adapter (TA) to aggregate historical information for temporal dependencies, and an Identity Adapter (IA) to leverage quality-aware contrastive learning for instance-level separability. Extensive experiments demonstrate that FDTA achieves state-of-the-art performance on multiple challenging MOT benchmarks, including DanceTrack, SportsMOT, and BFT, highlighting the effectiveness of our proposed discriminative embedding enhancement strategy. The code is available at this https URL.
Abstract (translated)
最近，端到端的多目标跟踪（MOT）方法通过在单一框架内统一检测和关联取得了显著进展。尽管这些方法具有强大的检测性能，但在关联准确性方面仍然存在相对较低的表现。经过详细分析，我们发现由共享DETR架构生成的对象嵌入表现出过高的对象间相似性，因为这种架构仅强调单帧内的类别级别区分。相比之下，跟踪要求跨帧在空间和时间连续性上进行实例级别的区别，而当前的端到端方法对优化对象嵌入不足。 为了解决这个问题，我们提出了FDTA（从检测到关联），这是一种显式的特征增强框架，它通过三个互补的角度增强了对象的区分度。具体来说，我们引入了空间适配器（SA）来整合深度感知线索以实现空间连续性、时间适配器（TA）来聚合历史信息以捕捉时间依赖关系以及身份适配器（IA），用于利用质量感知对比学习增强实例级别的可分离性。 广泛的实验表明，FDTA在多个具有挑战性的MOT基准测试中取得了最先进的性能，包括DanceTrack、SportsMOT和BFT。这突显了我们提出的区分式嵌入增强策略的有效性。代码可在[此处](https://this-url.com)获取。
URL
https://arxiv.org/abs/2512.02392
PDF
-
Physical ID-Transfer Attacks against Multi-Object Tracking via Adversarial Trajectory
2025-12-01 17:47:19Chenyi Wang, Yanmao Man, Raymond Muller, Ming Li, Z. Berkay Celik, Ryan Gerdes, Jonathan PetitAbstract
Multi-Object Tracking (MOT) is a critical task in computer vision, with applications ranging from surveillance systems to autonomous driving. However, threats to MOT algorithms have yet been widely studied. In particular, incorrect association between the tracked objects and their assigned IDs can lead to severe consequences, such as wrong trajectory predictions. Previous attacks against MOT either focused on hijacking the trackers of individual objects, or manipulating the tracker IDs in MOT by attacking the integrated object detection (OD) module in the digital domain, which are model-specific, non-robust, and only able to affect specific samples in offline datasets. In this paper, we present AdvTraj, the first online and physical ID-manipulation attack against tracking-by-detection MOT, in which an attacker uses adversarial trajectories to transfer its ID to a targeted object to confuse the tracking system, without attacking OD. Our simulation results in CARLA show that AdvTraj can fool ID assignments with 100% success rate in various scenarios for white-box attacks against SORT, which also have high attack transferability (up to 93% attack success rate) against state-of-the-art (SOTA) MOT algorithms due to their common design principles. We characterize the patterns of trajectories generated by AdvTraj and propose two universal adversarial maneuvers that can be performed by a human walker/driver in daily scenarios. Our work reveals under-explored weaknesses in the object association phase of SOTA MOT systems, and provides insights into enhancing the robustness of such systems.
Abstract (translated)
多目标跟踪（MOT）是计算机视觉中的一个关键任务，应用范围从监控系统到自动驾驶领域。然而，针对MOT算法的安全威胁尚未被广泛研究。尤其是，追踪对象与其分配ID之间的错误关联可能导致严重后果，比如轨迹预测错误。以往的攻击主要集中在劫持个别物体的追踪器或通过在网络域中攻击集成的对象检测（OD）模块来操纵MOT中的跟踪器ID上，这些方法针对特定模型、不具备鲁棒性，并且只能影响离线数据集中的特定样本。 在本文中，我们提出了AdvTraj——首个在线和物理级别的ID操控攻击，专门用于基于检测的多目标追踪系统。该攻击使用对抗轨迹让攻击者将其ID转移给选定的目标物体以混淆跟踪系统，而无需直接针对对象检测模块进行操作。我们在CARLA模拟环境中进行了实验，结果显示，对于SORT算法，在白盒攻击下AdvTraj可以以100%的成功率在各种场景中欺骗ID分配，并且由于现代顶级MOT算法的共同设计原则，该方法对其他SOTA MOT算法也具有高迁移性（最高达93%的成功率）。 我们描述了由AdvTraj生成的轨迹模式，并提出了两种通用对抗操作策略，这些策略可以被行人或驾驶员在日常场景中执行。我们的工作揭示了顶级MOT系统在目标关联阶段未被充分探索的安全弱点，并为提高此类系统的鲁棒性提供了见解。
URL
https://arxiv.org/abs/2512.01934
PDF
-
TransientTrack: Advanced Multi-Object Tracking and Classification of Cancer Cells with Transient Fluorescent Signals
2025-12-01 17:08:12Florian B\"urger, Martim Dias Gomes, Nica Gutu, Adri\'an E. Granada, No\'emie Moreau, Katarzyna BozekAbstract
Tracking cells in time-lapse videos is an essential technique for monitoring cell population dynamics at a single-cell level. Current methods for cell tracking are developed on videos with mostly single, constant signals and do not detect pivotal events such as cell death. Here, we present TransientTrack, a deep learning-based framework for cell tracking in multi-channel microscopy video data with transient fluorescent signals that fluctuate over time following processes such as the circadian rhythm of cells. By identifying key cellular events - mitosis (cell division) and apoptosis (cell death) our method allows us to build complete trajectories, including cell lineage information. TransientTrack is lightweight and performs matching on cell detection embeddings directly, without the need for quantification of tracking-specific cell features. Furthermore, our approach integrates Transformer Networks, multi-stage matching using all detection boxes, and the interpolation of missing tracklets with the Kalman Filter. This unified framework achieves strong performance across diverse conditions, effectively tracking cells and capturing cell division and death. We demonstrate the use of TransientTrack in an analysis of the efficacy of a chemotherapeutic drug at a single-cell level. The proposed framework could further advance quantitative studies of cancer cell dynamics, enabling detailed characterization of treatment response and resistance mechanisms. The code is available at this https URL.
Abstract (translated)
在时间推移视频中追踪细胞是一种监测单个细胞水平上细胞群体动态的重要技术。目前的细胞追踪方法主要针对信号单一且恒定的视频，无法检测到如细胞死亡这样的关键事件。在这里，我们提出了TransientTrack，这是一种基于深度学习的框架，用于多通道显微镜视频数据中的细胞追踪，该框架能够识别随时间波动的瞬时荧光信号（例如遵循细胞昼夜节律的过程）。通过识别关键的细胞事件——有丝分裂（细胞分裂）和凋亡（细胞死亡），我们的方法允许构建包括谱系信息在内的完整轨迹。TransientTrack轻量级且直接在检测嵌入上进行匹配，无需量化特定于追踪的细胞特征。此外，我们的方法整合了Transformer网络、多阶段匹配以及使用卡尔曼滤波器插补缺失片段的技术。这种统一框架在各种条件下均表现出色，能够有效跟踪细胞并捕捉到细胞分裂和死亡事件。 我们展示了TransientTrack在单细胞水平上分析化疗药物疗效的应用示例。所提出的框架可以进一步推进癌症细胞动力学的定量研究，使详细表征治疗反应及抗性机制成为可能。代码可在以下网址获取：[提供链接]
URL
https://arxiv.org/abs/2512.01885
PDF
-
BlinkBud: Detecting Hazards from Behind via Sampled Monocular 3D Detection on a Single Earbud
2025-12-01 07:25:17Yunzhe Li, Jiajun Yan, Yuzhou Wei, Kechen Liu, Yize Zhao, Chong Zhang, Hongzi Zhu, Li Lu, Shan Chang, Minyi GuoAbstract
Failing to be aware of speeding vehicles approaching from behind poses a huge threat to the road safety of pedestrians and cyclists. In this paper, we propose BlinkBud, which utilizes a single earbud and a paired phone to online detect hazardous objects approaching from behind of a user. The core idea is to accurately track visually identified objects utilizing a small number of sampled camera images taken from the earbud. To minimize the power consumption of the earbud and the phone while guaranteeing the best tracking accuracy, a novel 3D object tracking algorithm is devised, integrating both a Kalman filter based trajectory estimation scheme and an optimal image sampling strategy based on reinforcement learning. Moreover, the impact of constant user head movements on the tracking accuracy is significantly eliminated by leveraging the estimated pitch and yaw angles to correct the object depth estimation and align the camera coordinate system to the user's body coordinate system, respectively. We implement a prototype BlinkBud system and conduct extensive real-world experiments. Results show that BlinkBud is lightweight with ultra-low mean power consumptions of 29.8 mW and 702.6 mW on the earbud and smartphone, respectively, and can accurately detect hazards with a low average false positive ratio (FPR) and false negative ratio (FNR) of 4.90% and 1.47%, respectively.
Abstract (translated)
未能察觉后方快速接近的车辆对行人和骑自行车者构成了巨大的道路安全威胁。在本文中，我们提出了BlinkBud系统，该系统利用单个耳塞和配对手机来实时检测用户后方接近的危险物体。核心思想是通过少量从耳塞拍摄的照片样本准确追踪视觉识别的对象。 为了在保证最佳跟踪精度的同时最小化耳塞和手机的能耗，设计了一种新型的3D对象跟踪算法，该算法结合了基于卡尔曼滤波器的轨迹估计方案以及一种基于强化学习的最佳图像采样策略。此外，通过利用预估的角度（俯仰角和偏航角）来校正物体深度估算并将相机坐标系对准用户的体感坐标系，显著消除了持续头部运动对跟踪精度的影响。 我们实现了一个原型BlinkBud系统，并进行了广泛的实地实验。结果表明，BlinkBud轻量级且能耗极低，在耳塞和智能手机上的平均功率消耗分别为29.8毫瓦和702.6毫瓦；该系统能够以较低的平均假阳性率（FPR）和假阴性率（FNR），即4.90% 和 1.47%，准确地检测出危险。
URL
https://arxiv.org/abs/2512.01366
PDF
-
MANTA: Physics-Informed Generalized Underwater Object Tracking
2025-11-28 17:59:06Suhas Srinath, Hemang Jamadagni, Aditya Chadrasekar, Prathosh APAbstract
Underwater object tracking is challenging due to wavelength dependent attenuation and scattering, which severely distort appearance across depths and water conditions. Existing trackers trained on terrestrial data fail to generalize to these physics-driven degradations. We present MANTA, a physics-informed framework integrating representation learning with tracking design for underwater scenarios. We propose a dual-positive contrastive learning strategy coupling temporal consistency with Beer-Lambert augmentations to yield features robust to both temporal and underwater distortions. We further introduce a multi-stage pipeline augmenting motion-based tracking with a physics-informed secondary association algorithm that integrates geometric consistency and appearance similarity for re-identification under occlusion and drift. To complement standard IoU metrics, we propose Center-Scale Consistency (CSC) and Geometric Alignment Score (GAS) to assess geometric fidelity. Experiments on four underwater benchmarks (WebUOT-1M, UOT32, UTB180, UWCOT220) show that MANTA achieves state-of-the-art performance, improving Success AUC by up to 6 percent, while ensuring stable long-term generalized underwater tracking and efficient runtime.
Abstract (translated)
URL
https://arxiv.org/abs/2511.23405
PDF
-
DM$^3$T: Harmonizing Modalities via Diffusion for Multi-Object Tracking
2025-11-28 06:02:58Weiran Li, Yeqiang Liu, Yijie Wei, Mina Han, Qiannan Guo, Zhenbo LiAbstract
Multi-object tracking (MOT) is a fundamental task in computer vision with critical applications in autonomous driving and robotics. Multimodal MOT that integrates visible light and thermal infrared information is particularly essential for robust autonomous driving systems. However, effectively fusing these heterogeneous modalities is challenging. Simple strategies like concatenation or addition often fail to bridge the significant non-linear distribution gap between their feature representations, which can lead to modality conflicts and degrade tracking accuracy. Drawing inspiration from the connection between multimodal MOT and the iterative refinement in diffusion models, this paper proposes DM$^3$T, a novel framework that reformulates multimodal fusion as an iterative feature alignment process to generate accurate and temporally coherent object trajectories. Our approach performs iterative cross-modal harmonization through a proposed Cross-Modal Diffusion Fusion (C-MDF) module. In this process, features from both modalities provide mutual guidance, iteratively projecting them onto a shared, consistent feature manifold. This enables the learning of complementary information and achieves deeper fusion compared to conventional methods. Additionally, we introduce a plug-and-play Diffusion Refiner (DR) to enhance and refine the unified feature representation. To further improve tracking robustness, we design a Hierarchical Tracker that adaptively handles confidence estimation. DM$^3$T unifies object detection, state estimation, and data association into a comprehensive online tracking framework without complex post-processing. Extensive experiments on the VT-MOT benchmark demonstrate that our method achieves 41.7 HOTA, representing a 1.54% relative improvement over existing state-of-the-art methods. The code and models are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22896
PDF
-
Referring Video Object Segmentation with Cross-Modality Proxy Queries
2025-11-26 07:45:41Baoli Sun, Xinzhu Ma, Ning Wang, Zhihui Wang, Zhiyong WangAbstract
Referring video object segmentation (RVOS) is an emerging cross-modality task that aims to generate pixel-level maps of the target objects referred by given textual expressions. The main concept involves learning an accurate alignment of visual elements and language expressions within a semantic space. Recent approaches address cross-modality alignment through conditional queries, tracking the target object using a query-response based mechanism built upon transformer structure. However, they exhibit two limitations: (1) these conditional queries lack inter-frame dependency and variation modeling, making accurate target tracking challenging amid significant frame-to-frame variations; and (2) they integrate textual constraints belatedly, which may cause the video features potentially focus on the non-referred objects. Therefore, we propose a novel RVOS architecture called ProxyFormer, which introduces a set of proxy queries to integrate visual and text semantics and facilitate the flow of semantics between them. By progressively updating and propagating proxy queries across multiple stages of video feature encoder, ProxyFormer ensures that the video features are focused on the object of interest. This dynamic evolution also enables the establishment of inter-frame dependencies, enhancing the accuracy and coherence of object tracking. To mitigate high computational costs, we decouple cross-modality interactions into temporal and spatial dimensions. Additionally, we design a Joint Semantic Consistency (JSC) training strategy to align semantic consensus between the proxy queries and the combined video-text pairs. Comprehensive experiments on four widely used RVOS benchmarks demonstrate the superiority of our ProxyFormer to the state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21139
PDF
-
AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios
2025-11-26 04:44:27Chenglizhao Chen, Shaofeng Liang, Runwei Guan, Xiaolou Sun, Haocheng Zhao, Haiyun Jiang, Tao Huang, Henghui Ding, Qing-Long HanAbstract
Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21053
PDF
-
V$^{2}$-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondence
2025-11-25 22:06:30Jiancheng Pan, Runze Wang, Tianwen Qian, Mohammad Mahdi, Yanwei Fu, Xiangyang Xue, Xiaomeng Huang, Luc Van Gool, Danda Pani Paudel, Yuqian FuAbstract
Cross-view object correspondence, exemplified by the representative task of ego-exo object correspondence, aims to establish consistent associations of the same object across different viewpoints (e.g., ego-centric and exo-centric). This task poses significant challenges due to drastic viewpoint and appearance variations, making existing segmentation models, such as SAM2, non-trivial to apply directly. To address this, we present V^2-SAM, a unified cross-view object correspondence framework that adapts SAM2 from single-view segmentation to cross-view correspondence through two complementary prompt generators. Specifically, the Cross-View Anchor Prompt Generator (V^2-Anchor), built upon DINOv3 features, establishes geometry-aware correspondences and, for the first time, unlocks coordinate-based prompting for SAM2 in cross-view scenarios, while the Cross-View Visual Prompt Generator (V^2-Visual) enhances appearance-guided cues via a novel visual prompt matcher that aligns ego-exo representations from both feature and structural perspectives. To effectively exploit the strengths of both prompts, we further adopt a multi-expert design and introduce a Post-hoc Cyclic Consistency Selector (PCCS) that adaptively selects the most reliable expert based on cyclic consistency. Extensive experiments validate the effectiveness of V^2-SAM, achieving new state-of-the-art performance on Ego-Exo4D (ego-exo object correspondence), DAVIS-2017 (video object tracking), and HANDAL-X (robotic-ready cross-view correspondence).
Abstract (translated)
URL
https://arxiv.org/abs/2511.20886
PDF
