Abstract
With the advancement of drone technology, the volume of video data increases rapidly, creating an urgent need for efficient semantic retrieval. We are the first to systematically propose and study the drone video-text retrieval (DVTR) task. Drone videos feature overhead perspectives, strong structural homogeneity, and diverse semantic expressions of target combinations, which challenge existing cross-modal methods designed for ground-level views in effectively modeling their characteristics. Therefore, dedicated retrieval mechanisms tailored for drone scenarios are necessary. To address this issue, we propose a novel approach called Multi-Semantic Adaptive Mining (MSAM). MSAM introduces a multi-semantic adaptive learning mechanism, which incorporates dynamic changes between frames and extracts rich semantic information from specific scene regions, thereby enhancing the deep understanding and reasoning of drone video content. This method relies on fine-grained interactions between words and drone video frames, integrating an adaptive semantic construction module, a distribution-driven semantic learning term and a diversity semantic term to deepen the interaction between text and drone video modalities and improve the robustness of feature representation. To reduce the interference of complex backgrounds in drone videos, we introduce a cross-modal interactive feature fusion pooling mechanism that focuses on feature extraction and matching in target regions, minimizing noise effects. Extensive experiments on two self-constructed drone video-text datasets show that MSAM outperforms other existing methods in the drone video-text retrieval task. The source code and dataset will be made publicly available.
Abstract (translated)
随着无人机技术的进步,视频数据量迅速增加,这迫切需要高效的语义检索方法。我们首次系统地提出了并研究了无人机视频-文本检索(DVTR)任务。无人机视频具有俯视视角、较强的结构同质性和目标组合的多样化语义表达,这对现有旨在处理地面级视图的跨模态方法有效建模其特性构成了挑战。因此,针对无人机场景专门设计的检索机制是必要的。 为解决这一问题,我们提出了一种名为多语义自适应挖掘(MSAM)的新颖方法。MSAM引入了多语义自适应学习机制,该机制能够捕捉帧之间的动态变化,并从特定场景区域中提取丰富的语义信息,从而增强对无人机视频内容的深度理解和推理能力。这种方法依赖于文字与无人机视频帧之间精细级的互动,整合了一个自适应语义构建模块、一个以分布为驱动的语义学习项和多样化的语义项,深化了文本和无人机视频模态之间的交互,并提高了特征表示的鲁棒性。 为了减少复杂背景对无人机视频的影响,我们引入了一种跨模态互动特性融合池化机制,该机制专注于目标区域中的特征提取与匹配,以最小化噪声效应。在两个自建的无人机视频-文本数据集上的广泛实验表明,MSAM在无人机视频-文本检索任务中优于现有其他方法。源代码和数据集将会公开发布。
URL
https://arxiv.org/abs/2510.15470