Abstract
LiDAR-camera 3D multi-object tracking (MOT) combines rich visual semantics with accurate depth cues to improve trajectory consistency and tracking reliability. In practice, however, LiDAR and cameras operate at different sampling rates. To maintain temporal alignment, existing data pipelines usually synchronize heterogeneous sensor streams and annotate them at a reduced shared frequency, forcing most prior methods to perform spatial fusion only at synchronized timestamps through projection-based or learnable cross-sensor association. As a result, abundant asynchronous observations remain underexploited, despite their potential to support more frequent association and more robust trajectory estimation over short temporal intervals. To address this limitation, we propose Fusion-Poly, a spatial-temporal fusion framework for 3D MOT that integrates asynchronous LiDAR and camera data. Fusion-Poly associates trajectories with multi-modal observations at synchronized timestamps and with single-modal observations at asynchronous timestamps, enabling higher-frequency updates of motion and existence states. The framework contains three key components: a frequency-aware cascade matching module that adapts to synchronized and asynchronous frames according to available detection modalities; a frequency-aware trajectory estimation module that maintains trajectories through high-frequency motion prediction, differential updates, and confidence-calibrated lifecycle management; and a full-state observation alignment module that improves cross-modal consistency at synchronized timestamps by optimizing image-projection errors. On the nuScenes test set, Fusion-Poly achieves 76.5% AMOTA, establishing a new state of the art among tracking-by-detection 3D MOT methods. Extensive ablation studies further validate the effectiveness of each component. Code will be released.
Abstract (translated)
LiDAR-相机3D多目标跟踪(MOT)结合了丰富的视觉语义和准确的深度线索,以提高轨迹的一致性和跟踪的可靠性。然而,在实践中,由于LiDAR与摄像头的工作频率不同,现有数据管道通常会同步异构传感器流,并在较低的共享频率下进行标注,这迫使大多数先前的方法仅在同步时间戳通过基于投影或可学习跨传感器关联来进行空间融合。因此,尽管异步观测具有支持更频繁关联和短时区间内更强健轨迹估计的潜力,但大量未充分利用。 为了解决这一限制,我们提出了Fusion-Poly,这是一种用于3D MOT的空间-时间融合框架,能够整合异步LiDAR和相机数据。Fusion-Poly在同步时间戳处将轨迹与多模态观察相关联,并在异步时间戳处仅使用单模态观察进行关联,从而实现了运动和存在状态的更高频率更新。 该框架包含三个关键组件: 1. **频率感知级联匹配模块**:根据可用检测模式自适应地处理同步帧和异步帧。 2. **频率感知轨迹估计模块**:通过高频运动预测、差分更新以及基于置信度的生命周期管理来维持轨迹。 3. **全状态观测对齐模块**:在同步时间戳处优化图像投影误差,提高跨模式一致性。 在nuScenes测试集中,Fusion-Poly达到了76.5%的AMOTA(平均多目标跟踪准确率),确立了其在基于检测的3D MOT方法中的最新技术水平。广泛的消融研究进一步验证了每个组件的有效性。代码将公开发布。
URL
https://arxiv.org/abs/2603.08199