Paper Reading AI Learner

WAVE-DETR Multi-Modal Visible and Acoustic Real-Life Drone Detector

2025-09-11 21:18:32
Razvan Stefanescu, Ethan Oh, Ruben Vazquez, Chris Mesterharm, Constantin Serban, Ritu Chadha

Abstract

We introduce a multi-modal WAVE-DETR drone detector combining visible RGB and acoustic signals for robust real-life UAV object detection. Our approach fuses visual and acoustic features in a unified object detector model relying on the Deformable DETR and Wav2Vec2 architectures, achieving strong performance under challenging environmental conditions. Our work leverage the existing Drone-vs-Bird dataset and the newly generated ARDrone dataset containing more than 7,500 synchronized images and audio segments. We show how the acoustic information is used to improve the performance of the Deformable DETR object detector on the real ARDrone dataset. We developed, trained and tested four different fusion configurations based on a gated mechanism, linear layer, MLP and cross attention. The Wav2Vec2 acoustic embeddings are fused with the multi resolution feature mappings of the Deformable DETR and enhance the object detection performance over all drones dimensions. The best performer is the gated fusion approach, which improves the mAP of the Deformable DETR object detector on our in-distribution and out-of-distribution ARDrone datasets by 11.1% to 15.3% for small drones across all IoU thresholds between 0.5 and 0.9. The mAP scores for medium and large drones are also enhanced, with overall gains across all drone sizes ranging from 3.27% to 5.84%.

Abstract (translated)

我们介绍了一种结合可见光RGB和声学信号的多模态WAVE-DETR无人机检测器,用于在现实生活中进行稳健的UAV(无人飞行器)目标检测。我们的方法在一个统一的目标检测模型中融合了视觉和听觉特征,该模型基于Deformable DETR和Wav2Vec2架构,在具有挑战性的环境条件下取得了卓越的表现。我们的工作利用现有的Drone-vs-Bird数据集以及新生成的包含超过7,500个同步图像和音频片段的ARDrone数据集。 我们展示了声学信息是如何在真实世界的ARDrone数据集中提高Deformable DETR目标检测器性能的。基于门控机制、线性层、多层感知机(MLP)和交叉注意力,我们开发并测试了四种不同的融合配置。Wav2Vec2声学嵌入与Deformable DETR的多分辨率特征映射相融合,并在所有无人机尺寸上增强了目标检测性能。 表现最佳的是门控融合方法,在我们的分布内和分布外ARDrone数据集中,对于小型无人机而言,它使Deformable DETR目标检测器在IoU(交并比)阈值为0.5至0.9的所有阈值下的mAP提高了11.1%到15.3%。中型和大型无人机的mAP评分也得到了增强,在所有无人机尺寸上的总体改进范围从3.27%到5.84%不等。

URL

https://arxiv.org/abs/2509.09859

PDF

https://arxiv.org/pdf/2509.09859.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot