Abstract
Multi-view 3D geometry networks offer a powerful prior but are prohibitively slow for real-time applications. We propose a novel way to adapt them for online use, enabling real-time 6-DoF pose tracking and online reconstruction of objects and scenes from monocular RGB videos. Our method rapidly selects and manages a set of images as keyframes to map a scene or object via $\pi^3$ with full bidirectional attention. We then cache the global self-attention block's key-value (KV) pairs and use them as the sole scene representation for online tracking. This allows for up to $15\times$ speedup during inference without the fear of drift or catastrophic forgetting. Our caching strategy is model-agnostic and can be applied to other off-the-shelf multi-view networks without retraining. We demonstrate KV-Tracker on both scene-level tracking and the more challenging task of on-the-fly object tracking and reconstruction without depth measurements or object priors. Experiments on the TUM RGB-D, 7-Scenes, Arctic and OnePose datasets show the strong performance of our system while maintaining high frame-rates up to ${\sim}27$ FPS.
Abstract (translated)
多视角3D几何网络提供了一种强大的先验知识，但其运行速度过慢，无法满足实时应用的需求。我们提出一种新的方法来适应这些模型以实现在线使用，从而能够从单目RGB视频中实现实时的6自由度姿态跟踪和场景或物体的在线重建。我们的方法快速选择并管理一组关键帧图像集，通过$\pi^3$（全双向注意力机制）映射场景或对象。随后，我们缓存全局自注意力模块的关键值对(KV对)，并将它们作为唯一的场景表示用于在线追踪。这种方法在推理阶段最多可实现15倍的速度提升，并且不会出现漂移或灾难性遗忘的问题。 我们的缓存策略是模型无关的，可以应用于其他现成的多视角网络而不需重新训练。我们在KV-Tracker上展示了这一方法对场景级别跟踪和更具挑战性的在线物体跟踪与重建任务的有效性（无需深度测量或对象先验）。在TUM RGB-D、7-Scenes、Arctic 和 OnePose数据集上的实验显示，我们的系统具有强大的性能，并且能保持高达约27 FPS的高帧率。
URL
https://arxiv.org/abs/2512.22581
