Abstract
We introduce a multi-modal WAVE-DETR drone detector combining visible RGB and acoustic signals for robust real-life UAV object detection. Our approach fuses visual and acoustic features in a unified object detector model relying on the Deformable DETR and Wav2Vec2 architectures, achieving strong performance under challenging environmental conditions. Our work leverage the existing Drone-vs-Bird dataset and the newly generated ARDrone dataset containing more than 7,500 synchronized images and audio segments. We show how the acoustic information is used to improve the performance of the Deformable DETR object detector on the real ARDrone dataset. We developed, trained and tested four different fusion configurations based on a gated mechanism, linear layer, MLP and cross attention. The Wav2Vec2 acoustic embeddings are fused with the multi resolution feature mappings of the Deformable DETR and enhance the object detection performance over all drones dimensions. The best performer is the gated fusion approach, which improves the mAP of the Deformable DETR object detector on our in-distribution and out-of-distribution ARDrone datasets by 11.1% to 15.3% for small drones across all IoU thresholds between 0.5 and 0.9. The mAP scores for medium and large drones are also enhanced, with overall gains across all drone sizes ranging from 3.27% to 5.84%.
Abstract (translated)
我们介绍了一种结合可见光RGB和声学信号的多模态WAVE-DETR无人机检测器,用于在现实生活中进行稳健的UAV(无人飞行器)目标检测。我们的方法在一个统一的目标检测模型中融合了视觉和听觉特征,该模型基于Deformable DETR和Wav2Vec2架构,在具有挑战性的环境条件下取得了卓越的表现。我们的工作利用现有的Drone-vs-Bird数据集以及新生成的包含超过7,500个同步图像和音频片段的ARDrone数据集。 我们展示了声学信息是如何在真实世界的ARDrone数据集中提高Deformable DETR目标检测器性能的。基于门控机制、线性层、多层感知机(MLP)和交叉注意力,我们开发并测试了四种不同的融合配置。Wav2Vec2声学嵌入与Deformable DETR的多分辨率特征映射相融合,并在所有无人机尺寸上增强了目标检测性能。 表现最佳的是门控融合方法,在我们的分布内和分布外ARDrone数据集中,对于小型无人机而言,它使Deformable DETR目标检测器在IoU(交并比)阈值为0.5至0.9的所有阈值下的mAP提高了11.1%到15.3%。中型和大型无人机的mAP评分也得到了增强,在所有无人机尺寸上的总体改进范围从3.27%到5.84%不等。
URL
https://arxiv.org/abs/2509.09859