Object_Tracking
-
PlugTrack: Multi-Perceptive Motion Analysis for Adaptive Fusion in Multi-Object Tracking
2025-11-17 08:03:11Seungjae Kim, SeungJoon Lee, MyeongAh ChoAbstract
Multi-object tracking (MOT) predominantly follows the tracking-by-detection paradigm, where Kalman filters serve as the standard motion predictor due to computational efficiency but inherently fail on non-linear motion patterns. Conversely, recent data-driven motion predictors capture complex non-linear dynamics but suffer from limited domain generalization and computational overhead. Through extensive analysis, we reveal that even in datasets dominated by non-linear motion, Kalman filter outperforms data-driven predictors in up to 34\% of cases, demonstrating that real-world tracking scenarios inherently involve both linear and non-linear patterns. To leverage this complementarity, we propose PlugTrack, a novel framework that adaptively fuses Kalman filter and data-driven motion predictors through multi-perceptive motion understanding. Our approach employs multi-perceptive motion analysis to generate adaptive blending factors. PlugTrack achieves significant performance gains on MOT17/MOT20 and state-of-the-art on DanceTrack without modifying existing motion predictors. To the best of our knowledge, PlugTrack is the first framework to bridge classical and modern motion prediction paradigms through adaptive fusion in MOT.
Abstract (translated)
URL
https://arxiv.org/abs/2511.13105
PDF
-
SOTFormer: A Minimal Transformer for Unified Object Tracking and Trajectory Prediction
2025-11-14 19:25:05Zhongping Dong, Pengyang Yu, Shuangjian Li, Liming Chen, Mohand Tahar KechadiAbstract
Accurate single-object tracking and short-term motion forecasting remain challenging under occlusion, scale variation, and temporal drift, which disrupt the temporal coherence required for real-time perception. We introduce \textbf{SOTFormer}, a minimal constant-memory temporal transformer that unifies object detection, tracking, and short-horizon trajectory prediction within a single end-to-end framework. Unlike prior models with recurrent or stacked temporal encoders, SOTFormer achieves stable identity propagation through a ground-truth-primed memory and a burn-in anchor loss that explicitly stabilizes initialization. A single lightweight temporal-attention layer refines embeddings across frames, enabling real-time inference with fixed GPU memory. On the Mini-LaSOT (20%) benchmark, SOTFormer attains 76.3 AUC and 53.7 FPS (AMP, 4.3 GB VRAM), outperforming transformer baselines such as TrackFormer and MOTRv2 under fast motion, scale change, and occlusion.
Abstract (translated)
URL
https://arxiv.org/abs/2511.11824
PDF
-
Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective
2025-11-14 16:56:01Nhat Chung, Taisei Hanyu, Toan Nguyen, Huy Le, Frederick Bumgarner, Duy Minh Ho Nguyen, Khoa Vo, Kashu Yamazaki, Chase Rainwater, Tung Kieu, Anh Nguyen, Ngan LeAbstract
As embodied agents operate in increasingly complex environments, the ability to perceive, track, and reason about individual object instances over time becomes essential, especially in tasks requiring sequenced interactions with visually similar objects. In these non-Markovian settings, key decision cues are often hidden in object-specific histories rather than the current scene. Without persistent memory of prior interactions (what has been interacted with, where it has been, or how it has changed) visuomotor policies may fail, repeat past actions, or overlook completed ones. To surface this challenge, we introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability. It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame. However, vision-language-action (VLA) models often struggle in such settings, with token scaling quickly becoming intractable even for tasks spanning just a few hundred frames. We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability. It maintains spatio-temporally consistent slot identities and leverages them through two mechanisms: (1) slot-state-space modeling for reconstructing short-term history, and (2) a relational encoder to align the input tokens with action decoding. Together, these components enable temporally grounded, context-aware action prediction. Experiments show Embodied-SlotSSM's baseline performance on LIBERO-Mem and general tasks, offering a scalable solution for non-Markovian reasoning in object-centric robotic policies.
Abstract (translated)
URL
https://arxiv.org/abs/2511.11478
PDF
-
Hand Held Multi-Object Tracking Dataset in American Football
2025-11-12 16:15:18Rintaro Otsubo, Kanta Sawafuji, Hideo SaitoAbstract
Multi-Object Tracking (MOT) plays a critical role in analyzing player behavior from videos, enabling performance evaluation. Current MOT methods are often evaluated using publicly available datasets. However, most of these focus on everyday scenarios such as pedestrian tracking or are tailored to specific sports, including soccer and basketball. Despite the inherent challenges of tracking players in American football, such as frequent occlusion and physical contact, no standardized dataset has been publicly available, making fair comparisons between methods difficult. To address this gap, we constructed the first dedicated detection and tracking dataset for the American football players and conducted a comparative evaluation of various detection and tracking methods. Our results demonstrate that accurate detection and tracking can be achieved even in crowded scenarios. Fine-tuning detection models improved performance over pre-trained models. Furthermore, when these fine-tuned detectors and re-identification models were integrated into tracking systems, we observed notable improvements in tracking accuracy compared to existing approaches. This work thus enables robust detection and tracking of American football players in challenging, high-density scenarios previously underserved by conventional methods.
Abstract (translated)
URL
https://arxiv.org/abs/2511.09455
PDF
-
DMSORT: An efficient parallel maritime multi-object tracking architecture for unmanned vessel platforms
2025-11-06 07:20:36Shengyu Tang, Zeyuan Lu, Jiazhi Dong, Changdong Yu, Xiaoyu Wang, Yaohui Lyu, Weihao XiaAbstract
Accurate perception of the marine environment through robust multi-object tracking (MOT) is essential for ensuring safe vessel navigation and effective maritime surveillance. However, the complicated maritime environment often causes camera motion and subsequent visual degradation, posing significant challenges to MOT. To address this challenge, we propose an efficient Dual-branch Maritime SORT (DMSORT) method for maritime MOT. The core of the framework is a parallel tracker with affine compensation, which incorporates an object detection and re-identification (ReID) branch, along with a dedicated branch for dynamic camera motion estimation. Specifically, a Reversible Columnar Detection Network (RCDN) is integrated into the detection module to leverage multi-level visual features for robust object detection. Furthermore, a lightweight Transformer-based appearance extractor (Li-TAE) is designed to capture global contextual information and generate robust appearance features. Another branch decouples platform-induced and target-intrinsic motion by constructing a projective transformation, applying platform-motion compensation within the Kalman filter, and thereby stabilizing true object trajectories. Finally, a clustering-optimized feature fusion module effectively combines motion and appearance cues to ensure identity consistency under noise, occlusion, and drift. Extensive evaluations on the Singapore Maritime Dataset demonstrate that DMSORT achieves state-of-the-art performance. Notably, DMSORT attains the fastest runtime among existing ReID-based MOT frameworks while maintaining high identity consistency and robustness to jitter and occlusion. Code is available at: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.04128
PDF
-
Multi-Object Tracking Retrieval with LLaVA-Video: A Training-Free Solution to MOT25-StAG Challenge
2025-11-05 10:01:31Yi Yang, Yiming Xu, Timo Kaiser, Hao Cheng, Bodo Rosenhahn, Michael Ying YangAbstract
In this report, we present our solution to the MOT25-Spatiotemporal Action Grounding (MOT25-StAG) Challenge. The aim of this challenge is to accurately localize and track multiple objects that match specific and free-form language queries, using video data of complex real-world scenes as input. We model the underlying task as a video retrieval problem and present a two-stage, zero-shot approach, combining the advantages of the SOTA tracking model FastTracker and Multi-modal Large Language Model LLaVA-Video. On the MOT25-StAG test set, our method achieves m-HIoU and HOTA scores of 20.68 and 10.73 respectively, which won second place in the challenge.
Abstract (translated)
URL
https://arxiv.org/abs/2511.03332
PDF
-
UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNs
2025-11-03 17:24:19Zhe Liu, Jinghua Hou, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, Xiang BaiAbstract
Although transformers have demonstrated remarkable capabilities across various domains, their quadratic attention mechanisms introduce significant computational overhead when processing long-sequence data. In this paper, we present a unified autonomous driving model, UniLION, which efficiently handles large-scale LiDAR point clouds, high-resolution multi-view images, and even temporal sequences based on the linear group RNN operator (i.e., performs linear RNN for grouped features). Remarkably, UniLION serves as a single versatile architecture that can seamlessly support multiple specialized variants (i.e., LiDAR-only, temporal LiDAR, multi-modal, and multi-modal temporal fusion configurations) without requiring explicit temporal or multi-modal fusion modules. Moreover, UniLION consistently delivers competitive and even state-of-the-art performance across a wide range of core tasks, including 3D perception (e.g., 3D object detection, 3D object tracking, 3D occupancy prediction, BEV map segmentation), prediction (e.g., motion prediction), and planning (e.g., end-to-end planning). This unified paradigm naturally simplifies the design of multi-modal and multi-task autonomous driving systems while maintaining superior performance. Ultimately, we hope UniLION offers a fresh perspective on the development of 3D foundation models in autonomous driving. Code is available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2511.01768
PDF
-
UniSOT: A Unified Framework for Multi-Modality Single Object Tracking
2025-11-03 10:23:53Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Xu Zhou, Feng WuAbstract
Single object tracking aims to localize target object with specific reference modalities (bounding box, natural language or both) in a sequence of specific video modalities (RGB, RGB+Depth, RGB+Thermal or RGB+Event.). Different reference modalities enable various human-machine interactions, and different video modalities are demanded in complex scenarios to enhance tracking robustness. Existing trackers are designed for single or several video modalities with single or several reference modalities, which leads to separate model designs and limits practical applications. Practically, a unified tracker is needed to handle various requirements. To the best of our knowledge, there is still no tracker that can perform tracking with these above reference modalities across these video modalities simultaneously. Thus, in this paper, we present a unified tracker, UniSOT, for different combinations of three reference modalities and four video modalities with uniform parameters. Extensive experimental results on 18 visual tracking, vision-language tracking and RGB+X tracking benchmarks demonstrate that UniSOT shows superior performance against modality-specific counterparts. Notably, UniSOT outperforms previous counterparts by over 3.0\% AUC on TNL2K across all three reference modalities and outperforms Un-Track by over 2.0\% main metric across all three RGB+X video modalities.
Abstract (translated)
URL
https://arxiv.org/abs/2511.01427
PDF
-
OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback
2025-11-01 11:28:05Kai Luo, Hao Shi, Kunyu Peng, Fei Teng, Sheng Wu, Kaiwei Wang, Kailun YangAbstract
This paper investigates Multi-Object Tracking (MOT) in panoramic imagery, which introduces unique challenges including a 360° Field of View (FoV), resolution dilution, and severe view-dependent distortions. Conventional MOT methods designed for narrow-FoV pinhole cameras generalize unsatisfactorily under these conditions. To address panoramic distortion, large search space, and identity ambiguity under a 360° FoV, OmniTrack++ adopts a feedback-driven framework that progressively refines perception with trajectory cues. A DynamicSSM block first stabilizes panoramic features, implicitly alleviating geometric distortion. On top of normalized representations, FlexiTrack Instances use trajectory-informed feedback for flexible localization and reliable short-term association. To ensure long-term robustness, an ExpertTrack Memory consolidates appearance cues via a Mixture-of-Experts design, enabling recovery from fragmented tracks and reducing identity drift. Finally, a Tracklet Management module adaptively switches between end-to-end and tracking-by-detection modes according to scene dynamics, offering a balanced and scalable solution for panoramic MOT. To support rigorous evaluation, we establish the EmboTrack benchmark, a comprehensive dataset for panoramic MOT that includes QuadTrack, captured with a quadruped robot, and BipTrack, collected with a bipedal wheel-legged robot. Together, these datasets span wide-angle environments and diverse motion patterns, providing a challenging testbed for real-world panoramic perception. Extensive experiments on JRDB and EmboTrack demonstrate that OmniTrack++ achieves state-of-the-art performance, yielding substantial HOTA improvements of +25.5% on JRDB and +43.07% on QuadTrack over the original OmniTrack. Datasets and code will be made publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00510
PDF
-
A Hybrid Approach for Visual Multi-Object Tracking
2025-10-28 13:22:24Toan Van Nguyen, Rasmus G. K. Christiansen, Dirk Kraft, Leon BodenhagenAbstract
This paper proposes a visual multi-object tracking method that jointly employs stochastic and deterministic mechanisms to ensure identifier consistency for unknown and time-varying target numbers under nonlinear dynamics. A stochastic particle filter addresses nonlinear dynamics and non-Gaussian noise, with support from particle swarm optimization (PSO) to guide particles toward state distribution modes and mitigate divergence through proposed fitness measures incorporating motion consistency, appearance similarity, and social-interaction cues with neighboring targets. Deterministic association further enforces identifier consistency via a proposed cost matrix incorporating spatial consistency between particles and current detections, detection confidences, and track penalties. Subsequently, a novel scheme is proposed for the smooth updating of target states while preserving their identities, particularly for weak tracks during interactions with other targets and prolonged occlusions. Moreover, velocity regression over past states provides trend-seed velocities, enhancing particle sampling and state updates. The proposed tracker is designed to operate flexibly for both pre-recorded videos and camera live streams, where future frames are unavailable. Experimental results confirm superior performance compared to state-of-the-art trackers. The source-code reference implementations of both the proposed method and compared-trackers are provided on GitHub: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.24410
PDF
-
GenTrack: A New Generation of Multi-Object Tracking
2025-10-28 13:13:20Toan Van Nguyen, Rasmus G. K. Christiansen, Dirk Kraft, Leon BodenhagenAbstract
This paper introduces a novel multi-object tracking (MOT) method, dubbed GenTrack, whose main contributions include: a hybrid tracking approach employing both stochastic and deterministic manners to robustly handle unknown and time-varying numbers of targets, particularly in maintaining target identity (ID) consistency and managing nonlinear dynamics, leveraging particle swarm optimization (PSO) with some proposed fitness measures to guide stochastic particles toward their target distribution modes, enabling effective tracking even with weak and noisy object detectors, integration of social interactions among targets to enhance PSO-guided particles as well as improve continuous updates of both strong (matched) and weak (unmatched) tracks, thereby reducing ID switches and track loss, especially during occlusions, a GenTrack-based redefined visual MOT baseline incorporating a comprehensive state and observation model based on space consistency, appearance, detection confidence, track penalties, and social scores for systematic and efficient target updates, and the first-ever publicly available source-code reference implementation with minimal dependencies, featuring three variants, including GenTrack Basic, PSO, and PSO-Social, facilitating flexible reimplementation. Experimental results have shown that GenTrack provides superior performance on standard benchmarks and real-world scenarios compared to state-of-the-art trackers, with integrated implementations of baselines for fair comparison. Potential directions for future work are also discussed. The source-code reference implementations of both the proposed method and compared-trackers are provided on GitHub: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.24399
PDF
-
PlanarTrack: A high-quality and challenging benchmark for large-scale planar object tracking
2025-10-27 14:18:13Yifan Jiao, Xinran Liu, Xiaoqiong Liu, Xiaohui Yuan, Heng Fan, Libo ZhangAbstract
Planar tracking has drawn increasing interest owing to its key roles in robotics and augmented reality. Despite recent great advancement, further development of planar tracking, particularly in the deep learning era, is largely limited compared to generic tracking due to the lack of large-scale platforms. To mitigate this, we propose PlanarTrack, a large-scale high-quality and challenging benchmark for planar tracking. Specifically, PlanarTrack consists of 1,150 sequences with over 733K frames, including 1,000 short-term and 150 new long-term videos, which enables comprehensive evaluation of short- and long-term tracking performance. All videos in PlanarTrack are recorded in unconstrained conditions from the wild, which makes PlanarTrack challenging but more realistic for real-world applications. To ensure high-quality annotations, each video frame is manually annotated by four corner points with multi-round meticulous inspection and refinement. To enhance target diversity of PlanarTrack, we only capture a unique target in one sequence, which is different from existing benchmarks. To our best knowledge, PlanarTrack is by far the largest and most diverse and challenging dataset dedicated to planar tracking. To understand performance of existing methods on PlanarTrack and to provide a comparison for future research, we evaluate 10 representative planar trackers with extensive comparison and in-depth analysis. Our evaluation reveals that, unsurprisingly, the top planar trackers heavily degrade on the challenging PlanarTrack, which indicates more efforts are required for improving planar tracking. Our data and results will be released at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.23368
PDF
-
GRAP-MOT: Unsupervised Graph-based Position Weighted Person Multi-camera Multi-object Tracking in a Highly Congested Space
2025-10-24 14:07:11Marek Socha, Micha{\l} Marczyk, Aleksander Kempski, Micha{\l} Cogiel, Pawe{\l} Foszner, Rados{\l}aw Zawiski, Micha{\l} StaniszewskiAbstract
GRAP-MOT is a new approach for solving the person MOT problem dedicated to videos of closed areas with overlapping multi-camera views, where person occlusion frequently occurs. Our novel graph-weighted solution updates a person's identification label online based on tracks and the person's characteristic features. To find the best solution, we deeply investigated all elements of the MOT process, including feature extraction, tracking, and community search. Furthermore, GRAP-MOT is equipped with a person's position estimation module, which gives additional key information to the MOT method, ensuring better results than methods without position data. We tested GRAP-MOT on recordings acquired in a closed-area model and on publicly available real datasets that fulfil the requirement of a highly congested space, showing the superiority of our proposition. Finally, we analyzed existing metrics used to compare MOT algorithms and concluded that IDF1 is more adequate than MOTA in such comparisons. We made our code, along with the acquired dataset, publicly available.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21482
PDF
-
Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers
2025-10-23 17:58:45Dean L Slack, G Thomas Hudson, Thomas Winterbottom, Noura Al MoubayedAbstract
Inspired by the performance and scalability of autoregressive large language models (LLMs), transformer-based models have seen recent success in the visual domain. This study investigates a transformer adaptation for video prediction with a simple end-to-end approach, comparing various spatiotemporal self-attention layouts. Focusing on causal modeling of physical simulations over time; a common shortcoming of existing video-generative approaches, we attempt to isolate spatiotemporal reasoning via physical object tracking metrics and unsupervised training on physical simulation datasets. We introduce a simple yet effective pure transformer model for autoregressive video prediction, utilizing continuous pixel-space representations for video prediction. Without the need for complex training strategies or latent feature-learning components, our approach significantly extends the time horizon for physically accurate predictions by up to 50% when compared with existing latent-space approaches, while maintaining comparable performance on common video quality metrics. In addition, we conduct interpretability experiments to identify network regions that encode information useful to perform accurate estimations of PDE simulation parameters via probing models, and find that this generalizes to the estimation of out-of-distribution simulation parameters. This work serves as a platform for further attention-based spatiotemporal modeling of videos via a simple, parameter efficient, and interpretable approach.
Abstract (translated)
受到自回归大型语言模型（LLMs）在性能和可扩展性方面的启发，基于Transformer的模型在视觉领域取得了近期的成功。本研究探讨了一种用于视频预测的Transformer适应方案，并采用简单的端到端方法比较了各种时空自我注意布局。专注于对物理仿真进行因果建模，这通常是现有视频生成方法的一个常见不足之处，我们尝试通过物理对象跟踪指标和在物理仿真数据集上的无监督训练来隔离时空推理。 我们引入了一种简单而有效的纯Transformer模型用于自回归视频预测，该模型利用连续的像素空间表示来进行视频预测。无需复杂的训练策略或潜在特征学习组件，我们的方法与现有的潜在空间方法相比，在物理准确性的预测时间范围上提高了50%以上，同时保持了在常见视频质量指标上的相当性能。 此外，我们进行了可解释性实验以识别网络区域，这些区域通过探针模型编码执行PDE（偏微分方程）仿真参数的精确估计所需的信息，并发现这可以推广到对分布外仿真参数的估计。这项工作提供了一个平台，用于进一步基于注意力机制进行视频的时空建模，采用简单、参数高效且可解释的方法。
URL
https://arxiv.org/abs/2510.20807
PDF
-
Radar-Camera Fused Multi-Object Tracking: Online Calibration and Common Feature
2025-10-23 17:54:57Lei Cheng, Siyang CaoAbstract
This paper presents a Multi-Object Tracking (MOT) framework that fuses radar and camera data to enhance tracking efficiency while minimizing manual interventions. Contrary to many studies that underutilize radar and assign it a supplementary role--despite its capability to provide accurate range/depth information of targets in a world 3D coordinate system--our approach positions radar in a crucial role. Meanwhile, this paper utilizes common features to enable online calibration to autonomously associate detections from radar and camera. The main contributions of this work include: (1) the development of a radar-camera fusion MOT framework that exploits online radar-camera calibration to simplify the integration of detection results from these two sensors, (2) the utilization of common features between radar and camera data to accurately derive real-world positions of detected objects, and (3) the adoption of feature matching and category-consistency checking to surpass the limitations of mere position matching in enhancing sensor association accuracy. To the best of our knowledge, we are the first to investigate the integration of radar-camera common features and their use in online calibration for achieving MOT. The efficacy of our framework is demonstrated by its ability to streamline the radar-camera mapping process and improve tracking precision, as evidenced by real-world experiments conducted in both controlled environments and actual traffic scenarios. Code is available at this https URL
Abstract (translated)
这篇论文提出了一种多目标跟踪（MOT）框架，该框架融合了雷达和相机数据以提高跟踪效率，并减少人工干预。与许多将雷达的作用局限于辅助角色且未能充分利用其提供世界三维坐标系统中目标精确距离/深度信息的能力的研究不同，我们的方法使雷达发挥了关键作用。同时，本文利用共同特征来实现在线校准，以便自主关联来自雷达和相机的检测结果。 本工作的主要贡献包括： 1. 开发了一种基于雷达-相机融合的MOT框架，并通过利用在线雷达-相机校准简化了这两种传感器检测结果的整合。 2. 利用雷达与相机数据之间的共同特征来准确推导出被检测对象在真实世界中的位置。 3. 采用特征匹配和类别一致性检查，超越仅基于位置匹配的方法限制，在提高传感器关联准确性方面取得了进步。 据我们所知，这是我们首次研究雷达-相机共性特征的集成及其在线校准的应用，以实现MOT。我们的框架的有效性通过其能够简化雷达-相机映射过程并提升跟踪精度的能力得到了证明，这一点在控制环境和实际交通场景中的真实世界实验中都得到了验证。 代码可在[提供的URL]获取。
URL
https://arxiv.org/abs/2510.20794
PDF
-
FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking
2025-10-22 19:25:01Martha Teiko Teye, Ori Maoz, Matthias RottmannAbstract
We propose FutrTrack, a modular camera-LiDAR multi-object tracking framework that builds on existing 3D detectors by introducing a transformer-based smoother and a fusion-driven tracker. Inspired by query-based tracking frameworks, FutrTrack employs a multimodal two-stage transformer refinement and tracking pipeline. Our fusion tracker integrates bounding boxes with multimodal bird's-eye-view (BEV) fusion features from multiple cameras and LiDAR without the need for an explicit motion model. The tracker assigns and propagates identities across frames, leveraging both geometric and semantic cues for robust re-identification under occlusion and viewpoint changes. Prior to tracking, we refine sequences of bounding boxes with a temporal smoother over a moving window to refine trajectories, reduce jitter, and improve spatial consistency. Evaluated on nuScenes and KITTI, FutrTrack demonstrates that query-based transformer tracking methods benefit significantly from multimodal sensor features compared with previous single-sensor approaches. With an aMOTA of 74.7 on the nuScenes test set, FutrTrack achieves strong performance on 3D MOT benchmarks, reducing identity switches while maintaining competitive accuracy. Our approach provides an efficient framework for improving transformer-based trackers to compete with other neural-network-based methods even with limited data and without pretraining.
Abstract (translated)
我们提出了一种名为FutrTrack的模块化相机-激光雷达多目标跟踪框架，该框架在现有的3D检测器基础上引入了基于变换器的平滑器和融合驱动的跟踪器。受查询基元跟踪框架的启发，FutrTrack采用了一个跨模态两阶段变换器细化和跟踪流水线。我们的融合跟踪器将边界框与来自多个相机和激光雷达的多模态鸟瞰图（BEV）融合特征相结合，并不需要明确的运动模型。该跟踪器可以在帧之间分配并传播身份信息，利用几何和语义线索，在遮挡和视角变化的情况下实现稳健的重新识别。在进行追踪之前，我们通过在一个移动窗口上使用时序平滑器来细化边界框序列，以优化轨迹、减少抖动，并提高空间一致性。 在nuScenes和KITTI数据集上的评估表明，查询基元变换器跟踪方法从多模态传感器特征中受益匪浅，相较于之前的单传感器方法。FutrTrack在nuScenes测试集中达到了74.7的aMOTA（平均多目标跟踪准确率），展示了其在3D MOT基准测试中的强劲性能，在减少身份切换的同时保持了竞争力的精度。 我们的方法提供了一个高效的框架，用于改进基于变换器的追踪器，使其能够在数据有限且无预训练的情况下与其它神经网络方法竞争。
URL
https://arxiv.org/abs/2510.19981
PDF
-
HAD: Hierarchical Asymmetric Distillation to Bridge Spatio-Temporal Gaps in Event-Based Object Tracking
2025-10-22 13:15:13Yao Deng, Xian Zhong, Wenxuan Liu, Zhaofei Yu, Jingling Yuan, Tiejun HuangAbstract
RGB cameras excel at capturing rich texture details with high spatial resolution, whereas event cameras offer exceptional temporal resolution and a high dynamic range (HDR). Leveraging their complementary strengths can substantially enhance object tracking under challenging conditions, such as high-speed motion, HDR environments, and dynamic background interference. However, a significant spatio-temporal asymmetry exists between these two modalities due to their fundamentally different imaging mechanisms, hindering effective multi-modal integration. To address this issue, we propose {Hierarchical Asymmetric Distillation} (HAD), a multi-modal knowledge distillation framework that explicitly models and mitigates spatio-temporal asymmetries. Specifically, HAD proposes a hierarchical alignment strategy that minimizes information loss while maintaining the student network's computational efficiency and parameter compactness. Extensive experiments demonstrate that HAD consistently outperforms state-of-the-art methods, and comprehensive ablation studies further validate the effectiveness and necessity of each designed component. The code will be released soon.
Abstract (translated)
RGB相机擅长捕捉高空间分辨率的丰富纹理细节，而事件相机则提供卓越的时间分辨率和高动态范围（HDR）。利用这两种模态互补的优势可以显著增强在高速运动、HDR环境以及动态背景干扰等挑战条件下物体跟踪的效果。然而，由于它们的基本成像机制不同，这两种模式之间存在明显的时空不对称性，这阻碍了有效的多模态融合。为解决这一问题，我们提出了分层非对称蒸馏（Hierarchical Asymmetric Distillation, HAD），这是一个多模态知识蒸馏框架，它明确地建模和减轻了时空不对称性。具体而言，HAD提出了一种层次化对齐策略，在保持学生网络计算效率和参数精简的同时最小化信息损失。 大量的实验表明，HAD在各种基准测试中始终优于当前最先进的方法，并且全面的消融研究进一步验证了每个设计组件的有效性和必要性。代码即将发布。
URL
https://arxiv.org/abs/2510.19560
PDF
-
UniHPR: Unified Human Pose Representation via Singular Value Contrastive Learning
2025-10-21 21:06:51Zhongyu Jiang, Wenhao Chai, Lei Li, Zhuoran Zhou, Cheng-Yen Yang, Jenq-Neng HwangAbstract
In recent years, there has been a growing interest in developing effective alignment pipelines to generate unified representations from different modalities for multi-modal fusion and generation. As an important component of Human-Centric applications, Human Pose representations are critical in many downstream tasks, such as Human Pose Estimation, Action Recognition, Human-Computer Interaction, Object tracking, etc. Human Pose representations or embeddings can be extracted from images, 2D keypoints, 3D skeletons, mesh models, and lots of other modalities. Yet, there are limited instances where the correlation among all of those representations has been clearly researched using a contrastive paradigm. In this paper, we propose UniHPR, a unified Human Pose Representation learning pipeline, which aligns Human Pose embeddings from images, 2D and 3D human poses. To align more than two data representations at the same time, we propose a novel singular value-based contrastive learning loss, which better aligns different modalities and further boosts performance. To evaluate the effectiveness of the aligned representation, we choose 2D and 3D Human Pose Estimation (HPE) as our evaluation tasks. In our evaluation, with a simple 3D human pose decoder, UniHPR achieves remarkable performance metrics: MPJPE 49.9mm on the Human3.6M dataset and PA-MPJPE 51.6mm on the 3DPW dataset with cross-domain evaluation. Meanwhile, we are able to achieve 2D and 3D pose retrieval with our unified human pose representations in Human3.6M dataset, where the retrieval error is 9.24mm in MPJPE.
Abstract (translated)
近年来，人们对开发有效的对齐管道产生了越来越浓厚的兴趣，旨在从不同模态生成统一的表示形式，以进行多模态融合和生成。作为以人为中心的应用程序的重要组成部分，人体姿态表示在许多下游任务中至关重要，例如人体姿态估计、动作识别、人机交互、目标跟踪等。人体姿态表示或嵌入可以从图像、二维关键点、三维骨架、网格模型以及其他众多模态中提取出来。然而，在使用对比学习范式明确研究这些表示之间的相关性方面，案例相对有限。 在本文中，我们提出了UniHPR，一种统一的人体姿态表示学习管道，它对齐来自图像、二维和三维人体姿势的嵌入式表示。为了同时对齐超过两种数据表示形式，我们提出了一种基于奇异值的对比学习损失函数，这种方法能够更好地将不同模态进行对齐，并进一步提升性能表现。 为了评估对齐后的表示的有效性，我们将2D和3D人体姿态估计（HPE）作为评价任务。在我们的测试中，通过使用简单的三维人体姿势解码器，UniHPR取得了显著的性能指标：在Human3.6M数据集上实现了49.9毫米的MPJPE，在3DPW数据集中实现了51.6毫米的PA-MPJPE（跨域评估）。同时，我们能够在Human3.6M数据集上利用统一的人体姿态表示进行2D和3D姿势检索，并且在MPJPE指标下，检索误差仅为9.24毫米。
URL
https://arxiv.org/abs/2510.19078
PDF
-
Monitoring Horses in Stalls: From Object to Event Detection
2025-10-20 10:52:42Dmitrii Galimzianov, Viacheslav Vyshegorodtsev, Ivan NezhivykhAbstract
Monitoring the behavior of stalled horses is essential for early detection of health and welfare issues but remains labor-intensive and time-consuming. In this study, we present a prototype vision-based monitoring system that automates the detection and tracking of horses and people inside stables using object detection and multi-object tracking techniques. The system leverages YOLOv11 and BoT-SORT for detection and tracking, while event states are inferred based on object trajectories and spatial relations within the stall. To support development, we constructed a custom dataset annotated with assistance from foundation models CLIP and GroundingDINO. The system distinguishes between five event types and accounts for the camera's blind spots. Qualitative evaluation demonstrated reliable performance for horse-related events, while highlighting limitations in detecting people due to data scarcity. This work provides a foundation for real-time behavioral monitoring in equine facilities, with implications for animal welfare and stable management.
Abstract (translated)
监测停滞马的行为对于早期发现健康和福利问题至关重要，但这项工作既耗时又费力。在本研究中，我们提出了一种基于视觉的监控系统原型，该系统利用目标检测和多对象跟踪技术自动检测并追踪马匹及人员在马厩内的行为。此系统采用YOLOv11和BoT-SORT进行检测与跟踪，并根据物体轨迹和马厩内部的空间关系推断事件状态。为了支持系统的开发，我们构建了一个使用CLIP和GroundingDINO基础模型辅助注释的自定义数据集。该系统能够区分五种不同的事件类型，并考虑到摄像头盲区的影响。 定性评估表明，在与马匹相关的事件上，该系统表现可靠，但在因数据不足而难以检测人员时也存在局限性。这项工作为实现赛马设施中的实时行为监控奠定了基础，对动物福利和马厩管理具有重要意义。
URL
https://arxiv.org/abs/2510.17409
PDF
-
SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding
2025-10-16 15:16:51Tanveer Hannan, Shuaicong Wu, Mark Weber, Suprosanna Shit, Jindong Gu, Rajat Koner, Aljo\v{s}a O\v{s}ep, Laura Leal-Taix\'e, Thomas SeidlAbstract
Understanding fine-grained actions and accurately localizing their corresponding actors in space and time are fundamental capabilities for advancing next-generation AI systems, including embodied agents, autonomous platforms, and human-AI interaction frameworks. Despite recent progress in video understanding, existing methods predominantly address either coarse-grained action recognition or generic object tracking, thereby overlooking the challenge of jointly detecting and tracking multiple objects according to their actions while grounding them temporally. To address this gap, we introduce Spatio-temporal Video Action Grounding (SVAG), a novel task that requires models to simultaneously detect, track, and temporally localize all referent objects in videos based on natural language descriptions of their actions. To support this task, we construct SVAG-Bench, a large-scale benchmark comprising 688 videos, 19,590 annotated records, and 903 unique verbs, covering a diverse range of objects, actions, and real-world scenes. We further propose SVAGFormer, a baseline framework that adapts state of the art vision language models for joint spatial and temporal grounding, and introduce SVAGEval, a standardized evaluation toolkit for fair and reproducible benchmarking. Empirical results show that existing models perform poorly on SVAG, particularly in dense or complex scenes, underscoring the need for more advanced reasoning over fine-grained object-action interactions in long videos.
Abstract (translated)
理解细微动作并准确地在时间和空间上定位这些动作的执行者是推进下一代人工智能系统（包括具身代理、自主平台和人机交互框架）的基本能力。尽管近期视频理解方面取得了进展，现有的方法主要集中在粗粒度的动作识别或通用对象跟踪上，从而忽视了同时检测和根据动作追踪多个对象并将其时间定位的问题。为解决这一缺口，我们提出了时空视频行动接地（SVAG），这是一个要求模型基于自然语言描述的动作来同时检测、跟踪并在视频中临时定位所有相关对象的新任务。 为了支持这项任务，我们构建了SVAG-Bench，一个大规模基准测试集，包括688个视频、19,590条标注记录和903个独特的动词，涵盖了多样化的物体、动作和现实场景。此外，我们提出了SVAGFormer，一种基于最先进的视觉语言模型的基线框架，用于联合空间和时间定位，并引入了SVAGEval，一个标准化的评估工具包，用于公平和可重复的基准测试。 实证结果显示，现有的模型在SVAG任务上表现不佳，尤其是在密集或复杂的场景中，这强调了对长时间视频中的细粒度对象-动作交互进行更高级推理的需求。
URL
https://arxiv.org/abs/2510.13016
PDF
