Abstract
\noindent Memory has become the central mechanism enabling robust visual object tracking in modern segmentation-based frameworks. Recent methods built upon Segment Anything Model 2 (SAM2) have demonstrated strong performance by refining how past observations are stored and reused. However, existing approaches address memory limitations in a method-specific manner, leaving the broader design principles of memory in SAM-based tracking poorly understood. Moreover, it remains unclear how these memory mechanisms transfer to stronger, next-generation foundation models such as Segment Anything Model 3 (SAM3). In this work, we present a systematic memory-centric study of SAM-based visual object tracking. We first analyze representative SAM2-based trackers and show that most methods primarily differ in how short-term memory frames are selected, while sharing a common object-centric representation. Building on this insight, we faithfully reimplement these memory mechanisms within the SAM3 framework and conduct large-scale evaluations across ten diverse benchmarks, enabling a controlled analysis of memory design independent of backbone strength. Guided by our empirical findings, we propose a unified hybrid memory framework that explicitly decomposes memory into short-term appearance memory and long-term distractor-resolving memory. This decomposition enables the integration of existing memory policies in a modular and principled manner. Extensive experiments demonstrate that the proposed framework consistently improves robustness under long-term occlusion, complex motion, and distractor-heavy scenarios on both SAM2 and SAM3 backbones. Code is available at: this https URL. \textbf{This is a preprint. Some results are being finalized and may be updated in a future revision.}
Abstract (translated)
记忆已成为现代基于分割框架中实现稳健视觉目标跟踪的核心机制。最近，基于Segment Anything Model 2 (SAM2) 的方法通过改进过去观测值的存储和重用方式展示了强大的性能。然而，现有的方法以特定于方法的方式解决内存限制问题，这使得在基于SAM的追踪中的内存设计原则仍然不太清楚。此外，这些记忆机制如何转移到更强、下一代的基础模型如Segment Anything Model 3 (SAM3) 上仍然是未知的。 在这项工作中，我们提出了一个系统性的针对SAM基础视觉目标跟踪的记忆中心研究。首先，我们分析了代表性的基于SAM2的追踪器，并显示大多数方法主要在短期记忆帧的选择方式上有所不同，但共享着共同的目标为中心的表示法。基于这一见解，在SAM3框架内忠实重新实现了这些记忆机制，并在十个多样化的基准数据集上进行了大规模评估，从而能够独立于骨干网络强度地控制分析内存设计。依据我们的实证发现，我们提出了一种统一的混合记忆框架，该框架明确将内存分解为短期外观记忆和长期干扰解决记忆。这种分解使得以模块化且原理性的方式整合现有内存策略成为可能。 广泛的实验表明，在SAM2和SAM3基础网络上，所提出的框架在长时间遮挡、复杂运动及大量干扰场景下始终提高了稳健性。代码可在以下网址获得：此https URL。**这是一个预印本。一些结果正在最终确定中，并可能在未来修订版中更新。**
URL
https://arxiv.org/abs/2512.22624
