Paper Reading AI Learner

MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval

2025-10-17 09:26:28
Jinghao Huang, Yaxiong Chen, Ganchao Liu

Abstract

With the advancement of drone technology, the volume of video data increases rapidly, creating an urgent need for efficient semantic retrieval. We are the first to systematically propose and study the drone video-text retrieval (DVTR) task. Drone videos feature overhead perspectives, strong structural homogeneity, and diverse semantic expressions of target combinations, which challenge existing cross-modal methods designed for ground-level views in effectively modeling their characteristics. Therefore, dedicated retrieval mechanisms tailored for drone scenarios are necessary. To address this issue, we propose a novel approach called Multi-Semantic Adaptive Mining (MSAM). MSAM introduces a multi-semantic adaptive learning mechanism, which incorporates dynamic changes between frames and extracts rich semantic information from specific scene regions, thereby enhancing the deep understanding and reasoning of drone video content. This method relies on fine-grained interactions between words and drone video frames, integrating an adaptive semantic construction module, a distribution-driven semantic learning term and a diversity semantic term to deepen the interaction between text and drone video modalities and improve the robustness of feature representation. To reduce the interference of complex backgrounds in drone videos, we introduce a cross-modal interactive feature fusion pooling mechanism that focuses on feature extraction and matching in target regions, minimizing noise effects. Extensive experiments on two self-constructed drone video-text datasets show that MSAM outperforms other existing methods in the drone video-text retrieval task. The source code and dataset will be made publicly available.

Abstract (translated)

随着无人机技术的进步,视频数据量迅速增加,这迫切需要高效的语义检索方法。我们首次系统地提出了并研究了无人机视频-文本检索(DVTR)任务。无人机视频具有俯视视角、较强的结构同质性和目标组合的多样化语义表达,这对现有旨在处理地面级视图的跨模态方法有效建模其特性构成了挑战。因此,针对无人机场景专门设计的检索机制是必要的。 为解决这一问题,我们提出了一种名为多语义自适应挖掘(MSAM)的新颖方法。MSAM引入了多语义自适应学习机制,该机制能够捕捉帧之间的动态变化,并从特定场景区域中提取丰富的语义信息,从而增强对无人机视频内容的深度理解和推理能力。这种方法依赖于文字与无人机视频帧之间精细级的互动,整合了一个自适应语义构建模块、一个以分布为驱动的语义学习项和多样化的语义项,深化了文本和无人机视频模态之间的交互,并提高了特征表示的鲁棒性。 为了减少复杂背景对无人机视频的影响,我们引入了一种跨模态互动特性融合池化机制,该机制专注于目标区域中的特征提取与匹配,以最小化噪声效应。在两个自建的无人机视频-文本数据集上的广泛实验表明,MSAM在无人机视频-文本检索任务中优于现有其他方法。源代码和数据集将会公开发布。

URL

https://arxiv.org/abs/2510.15470

PDF

https://arxiv.org/pdf/2510.15470.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot