Abstract
Purpose: In this paper, we present a novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos, targeting localisation and tracking of critical anatomical structures and instruments. Our method addresses the challenges of costly pixel-level annotations and label inconsistencies inherent in existing datasets. Methods: Leveraging the inherent object localisation capabilities of pre-trained text-to-image diffusion models, we extract representative features from surgical frames without any training or fine-tuning. Our tracking framework uses these features, along with cross-frame interactions via an affinity matrix inspired by query-key-value attention, to ensure temporal continuity in the tracking process. Results: Through a pilot study, we first demonstrate that diffusion features exhibit superior object localisation and consistent semantics across different decoder levels and temporal frames. Later, we perform extensive experiments to validate the effectiveness of our approach, showcasing its superiority over competitors for the task of temporal object tracking. Specifically, we achieve a per-pixel classification accuracy of 79.19%, mean Jaccard Score of 56.20%, and mean F-Score of 79.48% on the publicly available CholeSeg8K dataset. Conclusion: Our work not only introduces a novel application of text-to-image diffusion models but also contributes to advancing the field of surgical video analysis, offering a promising avenue for accurate and cost-effective temporal object tracking in minimally invasive surgery videos.
Abstract (translated)
目的：本文提出了一种新颖的方法，用于腹腔镜胆囊切除术（LC）手术视频中的在线物体跟踪，旨在定位和追踪关键解剖结构和仪器。我们的方法解决了现有数据集中昂贵的像素级标注以及标签不一致的问题。 方法：通过利用预训练文本到图像扩散模型固有的对象定位能力，我们在没有任何训练或微调的情况下从手术帧中提取代表性特征。我们的跟踪框架使用这些特征，并结合跨帧互动（基于查询-键-值注意机制启发的亲和矩阵），以确保跟踪过程中的时间连续性。 结果：通过一项试点研究，我们首先证明了扩散特征在不同解码器层级和时间帧之间表现出色的对象定位能力和一致的语义。随后，我们进行了广泛的实验来验证我们的方法的有效性，并展示了其在时间对象跟踪任务中优于竞争对手的优势。具体而言，在公开可用的CholeSeg8K数据集上，我们实现了79.19%的每像素分类准确率、56.20%的平均Jaccard分数以及79.48%的平均F-Score。 结论：我们的工作不仅引入了文本到图像扩散模型的新应用方式，而且还为手术视频分析领域做出了贡献。它为微创手术视频中精确且成本效益高的时间对象跟踪提供了有前景的道路。
URL
https://arxiv.org/abs/2603.07839
