With the advancement of drone technology, the volume of video data increases rapidly, creating an urgent need for efficient semantic retrieval. We are the first to systematically propose and study the drone video-text retrieval (DVTR) task. Drone videos feature overhead perspectives, strong structural homogeneity, and diverse semantic expressions of target combinations, which challenge existing cross-modal methods designed for ground-level views in effectively modeling their characteristics. Therefore, dedicated retrieval mechanisms tailored for drone scenarios are necessary. To address this issue, we propose a novel approach called Multi-Semantic Adaptive Mining (MSAM). MSAM introduces a multi-semantic adaptive learning mechanism, which incorporates dynamic changes between frames and extracts rich semantic information from specific scene regions, thereby enhancing the deep understanding and reasoning of drone video content. This method relies on fine-grained interactions between words and drone video frames, integrating an adaptive semantic construction module, a distribution-driven semantic learning term and a diversity semantic term to deepen the interaction between text and drone video modalities and improve the robustness of feature representation. To reduce the interference of complex backgrounds in drone videos, we introduce a cross-modal interactive feature fusion pooling mechanism that focuses on feature extraction and matching in target regions, minimizing noise effects. Extensive experiments on two self-constructed drone video-text datasets show that MSAM outperforms other existing methods in the drone video-text retrieval task. The source code and dataset will be made publicly available.
随着无人机技术的进步,视频数据量迅速增加,这迫切需要高效的语义检索方法。我们首次系统地提出了并研究了无人机视频-文本检索(DVTR)任务。无人机视频具有俯视视角、较强的结构同质性和目标组合的多样化语义表达,这对现有旨在处理地面级视图的跨模态方法有效建模其特性构成了挑战。因此,针对无人机场景专门设计的检索机制是必要的。 为解决这一问题,我们提出了一种名为多语义自适应挖掘(MSAM)的新颖方法。MSAM引入了多语义自适应学习机制,该机制能够捕捉帧之间的动态变化,并从特定场景区域中提取丰富的语义信息,从而增强对无人机视频内容的深度理解和推理能力。这种方法依赖于文字与无人机视频帧之间精细级的互动,整合了一个自适应语义构建模块、一个以分布为驱动的语义学习项和多样化的语义项,深化了文本和无人机视频模态之间的交互,并提高了特征表示的鲁棒性。 为了减少复杂背景对无人机视频的影响,我们引入了一种跨模态互动特性融合池化机制,该机制专注于目标区域中的特征提取与匹配,以最小化噪声效应。在两个自建的无人机视频-文本数据集上的广泛实验表明,MSAM在无人机视频-文本检索任务中优于现有其他方法。源代码和数据集将会公开发布。
https://arxiv.org/abs/2510.15470
Unmanned Aerial Vehicles (UAVs) or drones, are increasingly used in search and rescue missions to detect human presence. Existing systems primarily leverage vision-based methods which are prone to fail under low-visibility or occlusion. Drone-based audio perception offers promise but suffers from extreme ego-noise that masks sounds indicating human presence. Existing datasets are either limited in diversity or synthetic, lacking real acoustic interactions, and there are no standardized setups for drone audition. To this end, we present DroneAudioset (The dataset is publicly available at this https URL under the MIT license), a comprehensive drone audition dataset featuring 23.5 hours of annotated recordings, covering a wide range of signal-to-noise ratios (SNRs) from -57.2 dB to -2.5 dB, across various drone types, throttles, microphone configurations as well as environments. The dataset enables development and systematic evaluation of noise suppression and classification methods for human-presence detection under challenging conditions, while also informing practical design considerations for drone audition systems, such as microphone placement trade-offs, and development of drone noise-aware audio processing. This dataset is an important step towards enabling design and deployment of drone-audition systems.
无人驾驶飞行器(UAV,即无人机)在搜索和救援任务中用于探测人类存在的情况日益增多。现有系统主要依赖视觉方法,在低能见度或遮挡的情况下容易失效。基于无人机的音频感知技术虽然很有前景,但因极端的自我噪音问题而无法捕捉到表明有人存在的声音信号。现有的数据集要么多样性有限,要么是合成数据,缺乏真实的声学交互,并且没有标准化的无人机听觉系统设置。 为此,我们介绍了DroneAudioset(该数据集可以在[此链接](https://opensource.org/licenses/MIT)下通过MIT许可公开获取),这是一个全面的无人机音频感知数据集,包含23.5小时注释录音,涵盖了从-57.2 dB到-2.5 dB范围内的各种信噪比(SNR),涉及多种无人机类型、油门设置、麦克风配置以及不同的环境。该数据集支持在挑战性条件下开发和系统化评估用于人类存在检测的降噪和分类方法,并且还为设计无人机听觉系统提供了实用的设计考虑,例如麦克风放置折中方案及开发具有无人机噪音感知能力的音频处理技术。此数据集是实现无人机听觉系统设计与部署的关键步骤。
https://arxiv.org/abs/2510.15383
Autonomous drone racing (ADR) systems have recently achieved champion-level performance, yet remain highly specific to drone racing. While end-to-end vision-based methods promise broader applicability, no system to date simultaneously achieves full sim-to-real transfer, onboard execution, and champion-level performance. In this work, we present SkyDreamer, to the best of our knowledge, the first end-to-end vision-based ADR policy that maps directly from pixel-level representations to motor commands. SkyDreamer builds on informed Dreamer, a model-based reinforcement learning approach where the world model decodes to privileged information only available during training. By extending this concept to end-to-end vision-based ADR, the world model effectively functions as an implicit state and parameter estimator, greatly improving interpretability. SkyDreamer runs fully onboard without external aid, resolves visual ambiguities by tracking progress using the state decoded from the world model's hidden state, and requires no extrinsic camera calibration, enabling rapid deployment across different drones without retraining. Real-world experiments show that SkyDreamer achieves robust, high-speed flight, executing tight maneuvers such as an inverted loop, a split-S and a ladder, reaching speeds of up to 21 m/s and accelerations of up to 6 g. It further demonstrates a non-trivial visual sim-to-real transfer by operating on poor-quality segmentation masks, and exhibits robustness to battery depletion by accurately estimating the maximum attainable motor RPM and adjusting its flight path in real-time. These results highlight SkyDreamer's adaptability to important aspects of the reality gap, bringing robustness while still achieving extremely high-speed, agile flight.
最近,自主无人机竞速(ADR)系统已经达到了冠军级别的表现,但这些系统仍高度特定于无人机竞速领域。虽然端到端视觉方法承诺更广泛的适用性,但迄今为止还没有一个系统能够同时实现从仿真环境到真实环境的完全迁移、在机上执行以及达到冠军级别性能。在这项工作中,我们介绍了SkyDreamer,在我们的知识范围内,它是第一个直接将像素级表示映射为电机指令的基于端到端视觉的ADR策略。SkyDreamer建立在知情梦想家的基础上,这是一种模型强化学习方法,其中世界模型解码成仅在训练期间可用的特权信息。通过将这一概念扩展到基于端到端视觉的ADR中,世界模型实际上充当了隐式状态和参数估计器,大大提高了可解释性。 SkyDreamer完全自主运行,无需外部支持,在飞行过程中利用从世界模型隐藏状态解码出的状态来解决视觉模糊问题,并且不需要外在相机校准,这使得它能够在不同无人机上快速部署而无需重新训练。真实世界的实验表明,SkyDreamer实现了稳健的高速飞行,执行诸如倒立环、分叉S和梯形等紧密机动动作,最高速度可达21米/秒,最大加速度可达6g。此外,通过在质量较差的分割掩码上操作并实时准确估计可达到的最大电机转速,并调整其飞行路径,SkyDreamer还展示了非平凡的视觉仿真到现实转移能力以及对电池耗尽的鲁棒性。 这些结果突显了SkyDreamer适应真实性和模拟环境之间重要差距方面的能力,在保持极高速度和敏捷飞行的同时实现稳健性能。
https://arxiv.org/abs/2510.14783
Reinforcement learning (RL) for the Markov Decision Process (MDP) has emerged in many security-related applications, such as autonomous driving, financial decisions, and drone/robot algorithms. In order to improve the robustness/defense of RL systems against adversaries, studying various adversarial attacks on RL systems is very important. Most previous work considered deterministic adversarial attack strategies in MDP, which the recipient (victim) agent can defeat by reversing the deterministic attacks. In this paper, we propose a provably ``invincible'' or ``uncounterable'' type of adversarial attack on RL. The attackers apply a rate-distortion information-theoretic approach to randomly change agents' observations of the transition kernel (or other properties) so that the agent gains zero or very limited information about the ground-truth kernel (or other properties) during the training. We derive an information-theoretic lower bound on the recipient agent's reward regret and show the impact of rate-distortion attacks on state-of-the-art model-based and model-free algorithms. We also extend this notion of an information-theoretic approach to other types of adversarial attack, such as state observation attacks.
针对马尔可夫决策过程(MDP)中的强化学习(RL),在许多安全相关应用中已经取得了广泛应用,例如自动驾驶、金融决策和无人机/机器人算法。为了提高面对对手时的鲁棒性和防御能力,研究各种对RL系统的对抗攻击是非常重要的。以往大多数工作都考虑了确定性的对抗性攻击策略,在这种情况下,接收者(受害者)可以通过逆转这些确定性攻击来击败它们。在这篇论文中,我们提出了一种理论上“无敌”或“不可计数”的类型对抗攻击方法。 攻击方采用速率失真信息理论的方法随机改变代理对转换核(或其他属性)的观察结果,使得代理在训练期间几乎无法获得有关真实内核(或其他属性)的有效信息。我们推导了接收者代理人回报遗憾的信息论下限,并展示了这些率失真攻击对最先进的基于模型和不基于模型算法的影响。此外,我们还扩展了这种信息理论方法到其他类型的对抗性攻击,例如状态观察攻击。 这种方法为防御强化学习系统面临的复杂安全挑战提供了一种新的视角,强调了在设计鲁棒的AI和机器学习解决方案时考虑随机、非确定性的干扰的重要性。
https://arxiv.org/abs/2510.13792
Drone technology is proliferating in many industries, including agriculture, logistics, defense, infrastructure, and environmental monitoring. Vision-based autonomy is one of its key enablers, particularly for real-world applications. This is essential for operating in novel, unstructured environments where traditional navigation methods may be unavailable. Autonomous drone racing has become the de facto benchmark for such systems. State-of-the-art research has shown that autonomous systems can surpass human-level performance in racing arenas. However, direct applicability to commercial and field operations is still limited as current systems are often trained and evaluated in highly controlled environments. In our contribution, the system's capabilities are analyzed within a controlled environment -- where external tracking is available for ground-truth comparison -- but also demonstrated in a challenging, uninstrumented environment -- where ground-truth measurements were never available. We show that our approach can match the performance of professional human pilots in both scenarios. We also publicly release the data from the flights carried out by our approach and a world-class human pilot.
无人机技术在农业、物流、国防、基础设施和环境监测等多个行业中广泛普及。基于视觉的自主性是其关键驱动力之一,特别是在实际应用中尤为重要。这在新颖且无结构化的环境中尤为必要,因为在这些环境中传统的导航方法可能不可用。自主无人机竞速已经成为此类系统事实上的基准测试标准。最新的研究表明,在赛车场中,自主系统可以超越人类水平的表现。然而,当前的系统通常是在高度控制的环境下进行训练和评估的,因此它们在商业和实地操作中的直接应用仍然有限。 在我们的贡献中,我们分析了系统的功能在一个受控环境中——在那里可以通过外部跟踪与真实数据对比——同时也展示了该系统在一个具有挑战性的、没有仪器支持的环境中运行的能力——在那里从未获得过真实测量数据。我们证明,在这两种情况下,我们的方法都能够匹配专业人类飞行员的表现水平。 此外,我们将公开发布由我们方法执行的飞行数据以及世界顶级人类飞行员的数据。
https://arxiv.org/abs/2510.13644
Multi-object tracking (MOT) from unmanned aerial vehicles (UAVs) presents unique challenges due to unpredictable object motion, frequent occlusions, and limited appearance cues inherent to aerial viewpoints. These issues are further exacerbated by abrupt UAV movements, leading to unreliable trajectory estimation and identity switches. Conventional motion models, such as Kalman filters or static sequence encoders, often fall short in capturing both linear and non-linear dynamics under such conditions. To tackle these limitations, we propose DMTrack, a deformable motion tracking framework tailored for UAV-based MOT. Our DMTrack introduces three key components: DeformMamba, a deformable state-space predictor that dynamically aggregates historical motion states for adaptive trajectory modeling; MotionGate, a lightweight gating module that fuses Kalman and Mamba predictions based on motion context and uncertainty; and an uncertainty-aware association strategy that enhances identity preservation by aligning motion trends with prediction confidence. Extensive experiments on the VisDrone-MOT and UAVDT benchmarks demonstrate that our DMTrack achieves state-of-the-art performance in identity consistency and tracking accuracy, particularly under high-speed and non-linear motion. Importantly, our method operates without appearance models and maintains competitive efficiency, highlighting its practicality for robust UAV-based tracking.
基于无人飞行器(UAV)的多目标跟踪(MOT)由于不可预测的目标运动、频繁遮挡以及空中视角固有的有限外观线索,面临着独特的挑战。这些问题还因无人机突然移动而加剧,导致轨迹估计不准确和身份切换。传统的运动模型,如卡尔曼滤波或静态序列编码器,在这种条件下捕捉线性和非线性动态时通常效果不佳。为了解决这些限制,我们提出了DMTrack,这是一个专为基于UAV的MOT设计的可变形运动跟踪框架。我们的DMTrack引入了三个关键组件:DeformMamba,一个可变形状态空间预测器,它根据历史运动状态进行自适应轨迹建模;MotionGate,一个轻量级门控模块,可以根据运动上下文和不确定性融合卡尔曼滤波和Mamba的预测;以及一种不确定性感知关联策略,通过将运动趋势与预测信心对齐来增强身份保持。在VisDrone-MOT和UAVDT基准测试中进行的广泛实验表明,我们的DMTrack在身份一致性和跟踪准确性方面达到了最先进的水平,特别是在高速度和非线性运动的情况下。重要的是,我们的方法不依赖于外观模型,并且保持了竞争性的效率,突显了其作为鲁棒基于UAV的跟踪的实际应用价值。
https://arxiv.org/abs/2510.17860
Feature detection is a common yet time-consuming module in Simultaneous Localization and Mapping (SLAM) implementations, which are increasingly deployed on power-constrained platforms, such as drones. Graphics Processing Units (GPUs) have been a popular accelerator for computer vision in general, and feature detection and SLAM in particular. On the other hand, System-on-Chips (SoCs) with integrated Field Programmable Gate Array (FPGA) are also widely available. This paper presents the first study of hardware-accelerated feature detectors considering a Visual SLAM (V-SLAM) pipeline. We offer new insights by comparing the best GPU-accelerated FAST, Harris, and SuperPoint implementations against the FPGA-accelerated counterparts on modern SoCs (Nvidia Jetson Orin and AMD Versal). The evaluation shows that when using a non-learning-based feature detector such as FAST and Harris, their GPU implementations, and the GPU-accelerated V-SLAM can achieve better run-time performance and energy efficiency than the FAST and Harris FPGA implementations as well as the FPGA-accelerated V-SLAM. However, when considering a learning-based detector such as SuperPoint, its FPGA implementation can achieve better run-time performance and energy efficiency (up to 3.1$\times$ and 1.4$\times$ improvements, respectively) than the GPU implementation. The FPGA-accelerated V-SLAM can also achieve comparable run-time performance compared to the GPU-accelerated V-SLAM, with better FPS in 2 out of 5 dataset sequences. When considering the accuracy, the results show that the GPU-accelerated V-SLAM is more accurate than the FPGA-accelerated V-SLAM in general. Last but not least, the use of hardware acceleration for feature detection could further improve the performance of the V-SLAM pipeline by having the global bundle adjustment module invoked less frequently without sacrificing accuracy.
特征检测是同步定位与建图(SLAM)实现中的一个常见但耗时的模块,而这些实现越来越多地部署在如无人机等能耗受限平台上。图形处理单元(GPU)一直是计算机视觉加速器的热门选择,尤其是对于特征检测和SLAM而言。另一方面,集成了现场可编程门阵列(FPGA)的片上系统(SoC)也广泛可用。本文首次研究了硬件加速特征检测器在视觉SLAM管道中的应用,并通过比较现代SoCs(如Nvidia Jetson Orin和AMD Versal)上的GPU加速FAST、Harris以及SuperPoint实现与对应的FPGA实现,提供了新的见解。 评估结果表明,在使用基于非学习的特征检测器(例如FAST和Harris)时,它们的GPU实现及GPU加速视觉SLAM能够比FAST和Harris的FPGA实现以及FPGA加速视觉SLAM获得更好的运行时间和能效。然而,当考虑SuperPoint这种基于学习的检测器时,其FPGA实现可以在运行时间性能和能效方面(分别高达3.1倍和1.4倍改进)优于GPU实现。在2个数据集中,FPGA加速视觉SLAM也能与GPU加速视觉SLAM达到相当的运行时间性能,并且在5个序列中的2个中FPS更优。 就准确性而言,结果表明,在一般情况下,GPU加速视觉SLAM比FPGA加速视觉SLAM更为准确。最后但同样重要的是,硬件加速特征检测器的应用可以通过较少地调用全局束调整模块进一步提升视觉SLAM管道的性能,同时不会牺牲精度。
https://arxiv.org/abs/2510.13546
Drone-based multi-object tracking is essential yet highly challenging due to small targets, severe occlusions, and cluttered backgrounds. Existing RGB-based tracking algorithms heavily depend on spatial appearance cues such as color and texture, which often degrade in aerial views, compromising reliability. Multispectral imagery, capturing pixel-level spectral reflectance, provides crucial cues that enhance object discriminability under degraded spatial conditions. However, the lack of dedicated multispectral UAV datasets has hindered progress in this domain. To bridge this gap, we introduce MMOT, the first challenging benchmark for drone-based multispectral multi-object tracking. It features three key characteristics: (i) Large Scale - 125 video sequences with over 488.8K annotations across eight categories; (ii) Comprehensive Challenges - covering diverse conditions such as extreme small targets, high-density scenarios, severe occlusions, and complex motion; and (iii) Precise Oriented Annotations - enabling accurate localization and reduced ambiguity under aerial perspectives. To better extract spectral features and leverage oriented annotations, we further present a multispectral and orientation-aware MOT scheme adapting existing methods, featuring: (i) a lightweight Spectral 3D-Stem integrating spectral features while preserving compatibility with RGB pretraining; (ii) an orientation-aware Kalman filter for precise state estimation; and (iii) an end-to-end orientation-adaptive transformer. Extensive experiments across representative trackers consistently show that multispectral input markedly improves tracking performance over RGB baselines, particularly for small and densely packed objects. We believe our work will advance drone-based multispectral multi-object tracking research. Our MMOT, code, and benchmarks are publicly available at this https URL.
基于无人机的多目标跟踪在面对小目标、严重遮挡和复杂背景时,虽然至关重要但也极具挑战性。现有的RGB(红绿蓝)基线追踪算法主要依赖于空间外观线索如颜色和纹理,但在航拍视角下这些线索往往会退化,从而影响可靠性。多光谱影像通过捕捉像素级的光谱反射率提供了关键信息,在空间条件较差的情况下可以增强对象的识别度。然而,缺乏专门针对无人机采集多光谱数据集的发展阻碍了该领域的进步。 为了填补这一空白,我们引入了MMOT,这是第一个用于基于无人机的多光谱多目标跟踪的挑战性基准测试平台。它具有三个关键特性:(i)大规模——包含125个视频序列及超过48.8万条注释;(ii)全面的挑战——涵盖了极端小目标、高密度场景、严重遮挡和复杂运动等多种情况;以及(iii)精确的方向标注——这使得在航拍视角下的精确定位成为可能,减少了不确定性。 为了更好地提取光谱特征并利用定向标注信息,我们进一步提出了一种多光谱感知的MOT方案,在保留与RGB预训练兼容性的前提下进行了现有方法的适应性改进。具体来说: (i) 一个轻量级的Spectral 3D-Stem,能够整合光谱特性; (ii) 一种基于方向的认知卡尔曼滤波器,用于准确的状态估计; (iii) 一种端到端的方向自适应变压器。 在多个代表性追踪器上进行广泛实验后显示,相较于RGB基线方法,多光谱输入显著提升了跟踪性能,尤其是在面对小型及密集目标时。我们相信这项工作将推进基于无人机的多光谱多目标跟踪研究领域的发展。我们的MMOT、代码和基准测试数据集已公开发布在以下网址:[https://this-url.com](https://this-url.com)(实际使用时请替换为正确的URL)。
https://arxiv.org/abs/2510.12565
In the multimedia domain, Infrared Small Target Detection (ISTD) plays a important role in drone-based multi-modality sensing. To address the dual challenges of cross-domain shift and heteroscedastic noise perturbations in ISTD, we propose a doubly wavelet-guided Invariance learning framework(Ivan-ISTD). In the first stage, we generate training samples aligned with the target domain using Wavelet-guided Cross-domain Synthesis. This wavelet-guided alignment machine accurately separates the target background through multi-frequency wavelet filtering. In the second stage, we introduce Real-domain Noise Invariance Learning, which extracts real noise characteristics from the target domain to build a dynamic noise library. The model learns noise invariance through self-supervised loss, thereby overcoming the limitations of distribution bias in traditional artificial noise modeling. Finally, we create the Dynamic-ISTD Benchmark, a cross-domain dynamic degradation dataset that simulates the distribution shifts encountered in real-world applications. Additionally, we validate the versatility of our method using other real-world datasets. Experimental results demonstrate that our approach outperforms existing state-of-the-art methods in terms of many quantitative metrics. In particular, Ivan-ISTD demonstrates excellent robustness in cross-domain scenarios. The code for this work can be found at: this https URL.
在多媒体领域,红外小目标检测(Infrared Small Target Detection,简称 ISTD)在基于无人机的多模态传感中扮演着重要角色。为了应对 ISTD 中跨域迁移和异方差噪声干扰这两大挑战,我们提出了一种双波let引导不变性学习框架(Ivan-ISTD)。该框架分为两个阶段: 第一阶段:利用 Wavelet-guided Cross-domain Synthesis 生成与目标领域对齐的训练样本。这种由波let引导的跨域合成机能够通过多频波let过滤器准确地分离出背景信息,从而达到目标领域的特征提取和背景分割的目的。 第二阶段:我们引入了 Real-domain Noise Invariance Learning(真实域噪声不变性学习),从目标领域中抽取真实的噪声特性来构建动态噪声库。模型通过自监督损失函数进行训练,使得其能够学会在面对真实环境中的不同噪声时保持性能的稳定性,从而克服传统人工噪声建模中分布偏差所带来的局限。 最后,我们创建了 Dynamic-ISTD Benchmark 数据集,这是一个模拟现实应用中跨域动态退化现象的数据集。此外,我们也使用其他的真实世界数据集验证了本方法的普适性。实验结果表明,与现有最先进的方法相比,我们的方法在许多定量指标上均表现出优越性能,在跨域场景中的鲁棒性尤为突出。 有关此工作的代码可在此 URL 下载:[请将此处替换为实际提供的链接]
https://arxiv.org/abs/2510.12241
Identifying drones and birds correctly is essential for keeping the skies safe and improving security systems. Using the VIP CUP 2025 dataset, which provides both RGB and infrared (IR) images, this study presents EGD-YOLOv8n, a new lightweight yet powerful model for object detection. The model improves how image features are captured and understood, making detection more accurate and efficient. It uses smart design changes and attention layers to focus on important details while reducing the amount of computation needed. A special detection head helps the model adapt to objects of different shapes and sizes. We trained three versions: one using RGB images, one using IR images, and one combining both. The combined model achieved the best accuracy and reliability while running fast enough for real-time use on common GPUs.
正确识别无人机和鸟类对于确保天空安全及提升安全系统至关重要。本研究利用VIP CUP 2025数据集(该数据集提供RGB图像与红外(IR)图像),提出了EGD-YOLOv8n这一新的轻量级且功能强大的目标检测模型。此模型改进了对图像特征的捕捉和理解,从而提高了检测的准确性和效率。它通过智能设计变更和注意力层专注于重要细节,并减少了所需的计算量。特别设计的目标检测头帮助模型适应不同形状和大小的对象。我们训练了三个版本:一个使用RGB图像,一个使用红外(IR)图像,还有一个结合两者的混合模型。最终,混合模型在精度与可靠性方面表现最佳,并且能够在通用GPU上以足够快的速度运行,适用于实时应用。
https://arxiv.org/abs/2510.10765
Unmanned aerial vehicles (UAVs) have become powerful platforms for real-time, high-resolution data collection, producing massive volumes of aerial videos. Efficient retrieval of relevant content from these videos is crucial for applications in urban management, emergency response, security, and disaster relief. While text-video retrieval has advanced in natural video domains, the UAV domain remains underexplored due to limitations in existing datasets, such as coarse and redundant captions. Thus, in this work, we construct the Drone Video-Text Match Dataset (DVTMD), which contains 2,864 videos and 14,320 fine-grained, semantically diverse captions. The annotations capture multiple complementary aspects, including human actions, objects, background settings, environmental conditions, and visual style, thereby enhancing text-video correspondence and reducing redundancy. Building on this dataset, we propose the Text-Conditioned Multi-granularity Alignment (TCMA) framework, which integrates global video-sentence alignment, sentence-guided frame aggregation, and word-guided patch alignment. To further refine local alignment, we design a Word and Patch Selection module that filters irrelevant content, as well as a Text-Adaptive Dynamic Temperature Mechanism that adapts attention sharpness to text type. Extensive experiments on DVTMD and CapERA establish the first complete benchmark for drone text-video retrieval. Our TCMA achieves state-of-the-art performance, including 45.5% R@1 in text-to-video and 42.8% R@1 in video-to-text retrieval, demonstrating the effectiveness of our dataset and method. The code and dataset will be released.
无人飞行器(UAVs)已成为实时、高分辨率数据采集的强大平台,生成了大量的空中视频。从这些视频中高效检索相关内容对于城市管理、应急响应、安全和灾害救援等应用至关重要。尽管在自然视频领域内文本-视频检索技术有所进步,但由于现有数据集存在粗略且冗余的描述等问题,UAV领域的研究尚处于探索阶段。 因此,在这项工作中,我们构建了无人机视频-文本匹配数据集(DVTMD),该数据集中包含了2,864个视频和14,320条精细、语义多样化的描述。这些标注涵盖了人类行为、物体、背景设置、环境条件及视觉风格等多个互补方面,从而增强了文本与视频之间的对应关系并减少了冗余性。 基于此数据集,我们提出了受文本调控的多粒度对齐框架(TCMA),该框架整合了全局视频-句子对齐、以句子为引导的帧聚合以及以单词为指导的补丁对齐。为了进一步优化局部对齐,我们设计了一个词语和补丁选择模块,用于过滤无关内容,并提出了一种基于文本适应性的动态温度机制,根据文本类型调整注意力强度。 在DVTMD和CapERA数据集上进行的广泛实验确立了首个完整的无人机文本-视频检索基准测试。我们的TCMA框架取得了最先进的性能,在文本到视频检索中达到45.5%的R@1,在视频到文本检索中达到42.8%的R@1,证明了我们所建立的数据集和方法的有效性。 代码与数据集将在后续发布。
https://arxiv.org/abs/2510.10180
The computer vision domain has greatly benefited from an abundance of data across many modalities to improve on various visual tasks. Recently, there has been a lot of focus on self-supervised pre-training methods through Masked Autoencoders (MAE) \cite{he2022masked,bachmann2022multimae}, usually used as a first step before optimizing for a downstream task, such as classification or regression. This is very useful as it doesn't require any manually labeled data. In this work, we introduce Probabilistic Hyper-Graphs using Masked Autoencoders (PHG-MAE): a novel model that unifies the classical work on neural graphs \cite{leordeanu2021semi} with the modern approach of masked autoencoders under a common theoretical framework. Through random masking of entire modalities, not just patches, the model samples from the distribution of hyper-edges on each forward pass. Additionally, the model adapts the standard MAE algorithm by combining pre-training and fine-tuning into a single training loop. Moreover, our approach enables the creation of inference-time ensembles which, through aggregation, boost the final prediction performance and consistency. Lastly, we show that we can apply knowledge distillation on top of the ensembles with little loss in performance, even with models that have fewer than 1M parameters. While our work mostly focuses on outdoor UAV scenes that contain multiple world interpretations and modalities, the same steps can be followed in other similar domains, such as autonomous driving or indoor robotics. In order to streamline the process of integrating external pre-trained experts for computer vision multi-modal multi-task learning (MTL) scenarios, we developed a data-pipeline software. Using this tool, we have created and released a fully-automated extension of the Dronescapes dataset. All the technical details, code and reproduction steps are publicly released.
计算机视觉领域从各种模态中获取的大量数据中受益匪浅,这有助于提高各类视觉任务的表现。最近,人们越来越多地关注通过掩码自编码器(MAE)进行自我监督预训练方法的应用,这些方法通常在针对分类或回归等下游任务优化之前使用。这种方法非常有用,因为它不需要任何人工标注的数据。在此项工作中,我们引入了基于掩码自编码器的概率超图模型(PHG-MAE),这是一种新颖的模型,它将经典的神经图研究工作与现代掩码自编码器方法在统一的理论框架下进行了融合。通过随机遮蔽整个模态而非只是图像块,在每次前向传播过程中,该模型从超边分布中采样数据。此外,该模型还对标准MAE算法进行了改进,将预训练和微调结合到了一个单一的训练循环中。我们的方法还能在推理阶段创建集成模型,通过聚合操作提高最终预测性能的一致性。最后,我们展示了可以在这些集合上应用知识蒸馏技术,并且即使对于参数数量少于1M的小型模型而言,性能也不会有显著损失。虽然我们的工作主要集中在含有多种世界解释和模态的室外无人机场景上,但类似的步骤也可以应用于其他类似领域,比如自动驾驶或室内机器人学。 为了简化计算机视觉多模态多任务学习(MTL)场景中集成外部预训练专家的过程,我们开发了一款数据管道软件。利用这一工具,我们创建并发布了Dronescapes数据集的全自动化扩展版本。所有技术细节、代码和复现步骤均已公开发布。
https://arxiv.org/abs/2510.10068
Small object detection in aerial images suffers from severe information degradation during feature extraction due to limited pixel representations, where shallow spatial details fail to align effectively with semantic information, leading to frequent misses and false positives. Existing FPN-based methods attempt to mitigate these losses through post-processing enhancements, but the reconstructed details often deviate from the original image information, impeding their fusion with semantic content. To address this limitation, we propose PRNet, a real-time detection framework that prioritizes the preservation and efficient utilization of primitive shallow spatial features to enhance small object representations. PRNet achieves this via two modules:the Progressive Refinement Neck (PRN) for spatial-semantic alignment through backbone reuse and iterative refinement, and the Enhanced SliceSamp (ESSamp) for preserving shallow information during downsampling via optimized rearrangement and convolution. Extensive experiments on the VisDrone, AI-TOD, and UAVDT datasets demonstrate that PRNet outperforms state-of-the-art methods under comparable computational constraints, achieving superior accuracy-efficiency trade-offs.
在航空图像中,小物体的检测面临严重的信息退化问题,在特征提取过程中,由于像素表示有限,浅层空间细节难以与语义信息有效对齐,导致频繁错过目标和产生假阳性。现有的基于FPN(特征金字塔网络)的方法试图通过后处理增强来缓解这些损失,但重建的细节往往偏离原始图像信息,阻碍了其与语义内容的有效融合。 为了解决这一限制,我们提出了PRNet,这是一种实时检测框架,优先考虑保留和高效利用基础浅层空间特性以增强小物体的表现。PRNet通过两个模块实现这一点:渐进式细化颈(Progressive Refinement Neck, PRN),用于通过复用主干网络和迭代细化来实现空间-语义对齐;以及增强的SliceSamp(Enhanced SliceSamp,ESSamp),在下采样过程中通过优化重组和卷积来保留浅层信息。 在VisDrone、AI-TOD和UAVDT数据集上的广泛实验表明,在类似的计算约束条件下,PRNet优于现有最先进的方法,并实现了更优的精度-效率权衡。
https://arxiv.org/abs/2510.09531
Modern agricultural operations increasingly rely on integrated monitoring systems that combine multiple data sources for farm optimization. Aerial drone-based animal health monitoring serves as a key component but faces limited data availability, compounded by scene-specific issues such as small, occluded, or partially visible animals. Transfer learning approaches often fail to address this limitation due to the unavailability of large datasets that reflect specific farm conditions, including variations in animal breeds, environments, and behaviors. Therefore, there is a need for developing a problem-specific, animal-focused data augmentation strategy tailored to these unique challenges. To address this gap, we propose an object-focused data augmentation framework designed explicitly for animal health monitoring in constrained data settings. Our approach segments animals from backgrounds and augments them through transformations and diffusion-based synthesis to create realistic, diverse scenes that enhance animal detection and monitoring performance. Our initial experiments demonstrate that our augmented dataset yields superior performance compared to our baseline models on the animal detection task. By generating domain-specific data, our method empowers real-time animal health monitoring solutions even in data-scarce scenarios, bridging the gap between limited data and practical applicability.
现代农业操作越来越依赖于结合多个数据来源的综合监测系统,以实现农场优化。基于无人机的动物健康监测是这一过程中的关键组成部分,但面临着有限的数据可用性问题,尤其是在特定场景中,如小型、被遮挡或部分可见的动物情况下尤为突出。由于缺乏能够反映具体农场条件的大规模数据集(包括不同品种、环境和行为的变化),迁移学习方法往往无法有效解决这些问题。因此,有必要开发一种专门针对这些独特挑战的问题特定、以动物为中心的数据增强策略。 为了解决这一缺口,我们提出了一种面向对象的、专为在受限数据环境中进行动物健康监测设计的数据增强框架。我们的方法通过分割动物与背景,并通过对动物应用变换和基于扩散的合成来生成现实且多样的场景,从而增强动物检测和监控性能。初步实验表明,在动物检测任务上,使用我们扩增后的数据集比基线模型表现出更优的效果。 通过生成特定领域的数据,这种方法能够在数据匮乏的情况下为实时动物健康监测解决方案提供支持,并弥合了有限数据与实际应用之间的差距。
https://arxiv.org/abs/2510.08955
Modeling the dynamics of micro-mobility vehicles (MMV) is becoming increasingly important for training autonomous vehicle systems and building urban traffic simulations. However, mainstream tools rely on variants of the Kinematic Bicycle Model (KBM) or mode-specific physics that miss tire slip, load transfer, and rider/vehicle lean. To our knowledge, no unified, physics-based model captures these dynamics across the full range of common MMVs and wheel layouts. We propose the "Generalized Micro-mobility Model" (GM3), a tire-level formulation based on the tire brush representation that supports arbitrary wheel configurations, including single/double track and multi-wheel platforms. We introduce an interactive model-agnostic simulation framework that decouples vehicle/layout specification from dynamics to compare the GM3 with the KBM and other models, consisting of fixed step RK4 integration, human-in-the-loop and scripted control, real-time trajectory traces and logging for analysis. We also empirically validate the GM3 on the Stanford Drone Dataset's deathCircle (roundabout) scene for biker, skater, and cart classes.
微移动车辆(MMV)的动力学建模对于训练自主车辆系统和构建城市交通模拟变得越来越重要。然而,主流工具依赖于Kinematic Bicycle Model (KBM)或特定模式的物理模型,这些模型忽略了轮胎滑移、负载转移以及骑行者/车辆倾斜等因素。据我们所知,目前没有一个统一的基于物理学的模型能够捕捉到不同种类MMV及其轮毂布局中的全部动力学特性。为此,我们提出了“广义微移动模型”(Generalized Micro-mobility Model, GM3),这是一个基于轮胎刷子表示法的轮胎级公式,支持任意轮配置,包括单/双轨和多轮平台。 为了评估GM3与其他现有模型之间的差异,我们引入了一个交互式、与特定模型无关的模拟框架。该框架将车辆和布局规范从动力学中解耦出来,并且通过固定步长RK4积分方法、人机环路(human-in-the-loop)以及脚本化控制来进行比较研究。此外,它还支持实时轨迹跟踪和日志记录以进行分析。 我们还在斯坦福无人机数据集的死亡圈场景(roundabout场景)上进行了GM3的实证验证,针对骑车人、滑板手和手推车类别进行了测试。
https://arxiv.org/abs/2510.07807
Underground water and wastewater pipelines are vital for city operations but plagued by anomalies like leaks and infiltrations, causing substantial water loss, environmental damage, and high repair costs. Conventional manual inspections lack efficiency, while dense sensor deployments are prohibitively expensive. In recent years, artificial intelligence has advanced rapidly and is increasingly applied to urban infrastructure. In this research, we propose an integrated AI and remote-sensor framework to address the challenge of leak detection in underground water pipelines, through deploying a sparse set of remote sensors to capture real-time flow and depth data, paired with HydroNet - a dedicated model utilizing pipeline attributes (e.g., material, diameter, slope) in a directed graph for higher-precision modeling. Evaluations on a real-world campus wastewater network dataset demonstrate that our system collects effective spatio-temporal hydraulic data, enabling HydroNet to outperform advanced baselines. This integration of edge-aware message passing with hydraulic simulations enables accurate network-wide predictions from limited sensor deployments. We envision that this approach can be effectively extended to a wide range of underground water pipeline networks.
地下供水和污水管道对于城市运营至关重要,但常常受到泄漏和其他异常情况的影响,这些情况导致了巨大的水资源损失、环境污染以及高昂的维修成本。传统的手动检查方法效率低下,而密集部署传感器则由于成本过高而不切实际。近年来,人工智能技术迅速发展,并越来越多地应用于城市基础设施领域。在这项研究中,我们提出了一种结合人工智能和远程传感框架的方法,以解决地下供水管道中的泄漏检测问题。该方法通过部署一组稀疏的远程传感器来捕获实时流量和深度数据,并与HydroNet模型相结合,后者利用管道属性(如材料、直径、坡度)在一个有向图中进行更高精度建模。 在对真实校园污水网络的数据集进行评估后,我们发现我们的系统能够有效收集空间和时间上的水力学数据,从而使HydroNet模型超越了现有的先进基准。通过将边缘感知消息传递与水力模拟相结合,即使在有限传感器部署的情况下也能实现准确的全网预测。我们认为这种方法可以有效地应用于广泛的地下供水管道网络中。
https://arxiv.org/abs/2510.06631
Artificial Potential Field (APF) methods are widely used for reactive flocking control, but they often suffer from challenges such as deadlocks and local minima, especially in the presence of obstacles. Existing solutions to address these issues are typically passive, leading to slow and inefficient collective navigation. As a result, many APF approaches have only been validated in obstacle-free environments or simplified, pseudo 3D simulations. This paper presents GO-Flock, a hybrid flocking framework that integrates planning with reactive APF-based control. GO-Flock consists of an upstream Perception Module, which processes depth maps to extract waypoints and virtual agents for obstacle avoidance, and a downstream Collective Navigation Module, which applies a novel APF strategy to achieve effective flocking behavior in cluttered environments. We evaluate GO-Flock against passive APF-based approaches to demonstrate their respective merits, such as their flocking behavior and the ability to overcome local minima. Finally, we validate GO-Flock through obstacle-filled environment and also hardware-in-the-loop experiments where we successfully flocked a team of nine drones, six physical and three virtual, in a forest environment.
人工势场(APF)方法在反应式群集控制中被广泛应用,但它们通常会遇到诸如死锁和局部极小值等问题,尤其是在有障碍物的情况下。现有的解决方案通常是被动的,导致集体导航缓慢且效率低下。因此,许多APF方法仅在无障碍环境中或简化的伪3D模拟中得到了验证。 本文介绍了GO-Flock,这是一种将规划与基于反应式APF控制结合在一起的混合群集框架。GO-Flock由一个上游感知模块和一个下游集体导航模块组成:上游感知模块处理深度图以提取用于避障的航路点和虚拟代理;下游集体导航模块则应用一种新的APF策略,以便在拥挤环境中实现有效的群体行为。 我们通过与被动式的APF方法进行对比评估,来展示GO-Flock及其相关技术各自的优势,包括其群集行为以及克服局部极小值的能力。最后,我们在充满障碍物的环境和硬件在环实验中验证了GO-Flock的有效性,在森林环境中成功地将九架无人机(六架物理无人机和三架虚拟无人机)进行了群组飞行。 通过这种方法,GO-Flock旨在解决传统APF方法中存在的问题,并展示了一个更加灵活且适应性强的群体导航解决方案。
https://arxiv.org/abs/2510.05553
Accurate pedestrian trajectory prediction is crucial for ensuring safety and efficiency in autonomous driving and human-robot interaction scenarios. Earlier studies primarily utilized sufficient observational data to predict future trajectories. However, in real-world scenarios, such as pedestrians suddenly emerging from blind spots, sufficient observational data is often unavailable (i.e. momentary trajectory), making accurate prediction challenging and increasing the risk of traffic accidents. Therefore, advancing research on pedestrian trajectory prediction under extreme scenarios is critical for enhancing traffic safety. In this work, we propose a novel framework termed Diffusion^2, tailored for momentary trajectory prediction. Diffusion^2 consists of two sequentially connected diffusion models: one for backward prediction, which generates unobserved historical trajectories, and the other for forward prediction, which forecasts future trajectories. Given that the generated unobserved historical trajectories may introduce additional noise, we propose a dual-head parameterization mechanism to estimate their aleatoric uncertainty and design a temporally adaptive noise module that dynamically modulates the noise scale in the forward diffusion process. Empirically, Diffusion^2 sets a new state-of-the-art in momentary trajectory prediction on ETH/UCY and Stanford Drone datasets.
准确的行人轨迹预测对于确保自动驾驶和人机交互场景中的安全性和效率至关重要。早期的研究主要利用充分的观测数据来预测未来的轨迹。然而,在现实世界中,例如行人在盲区突然出现的情况下,往往无法获得足够的观测数据(即瞬间轨迹),这使得精确预测变得具有挑战性,并增加了交通事故的风险。因此,推进极端情况下的行人轨迹预测研究对于提高交通安全至关重要。 为此,我们提出了一种新的框架称为Diffusion^2,该框架专为瞬间轨迹预测设计。Diffusion^2包含两个顺序连接的扩散模型:一个用于反向预测,生成未观察的历史轨迹;另一个用于正向预测,预测未来的轨迹。鉴于生成的未观察历史轨迹可能会引入额外噪声,我们提出了一种双头参数化机制来估计其固有不确定性,并设计了一个时间自适应噪声模块,在正向扩散过程中动态调节噪声尺度。 从实验结果来看,Diffusion^2在ETH/UCY和斯坦福无人机数据集上的瞬间轨迹预测方面达到了新的最先进水平。
https://arxiv.org/abs/2510.04365
This paper considers the path planning problem for autonomous exploration of an unknown environment using multiple heterogeneous robots such as drones, wheeled, and legged robots, which have different capabilities to traverse complex terrains. A key challenge there is to intelligently allocate the robots to the unknown areas to be explored and determine the visiting order of those spaces subject to traversablity constraints, which leads to a large scale constrained optimization problem that needs to be quickly and iteratively solved every time when new space are explored. To address the challenge, we propose HEHA (Hierarchical Exploration with Heterogeneous Agents) by leveraging a recent hierarchical method that decompose the exploration into global planning and local planning. The major contribution in HEHA is its global planning, where we propose a new routing algorithm PEAF (Partial Anytime Focal search) that can quickly find bounded sub-optimal solutions to minimize the maximum path length among the agents subject to traversability constraints. Additionally, the local planner in HEHA also considers heterogeneity to avoid repeated and duplicated exploration among the robots. The experimental results show that, our HEHA can reduce up to 30% of the exploration time than the baselines.
本文探讨了使用多种异构机器人(如无人机、轮式和腿足机器人)在未知环境中进行自主探索的路径规划问题。这些机器人的能力各异,能够穿越复杂的地形。其中的关键挑战是如何智能地分配机器人去探索未被了解的区域,并确定它们访问顺序以满足通行性约束条件,这导致了一个需要快速迭代求解的大规模受限优化问题,每当有新的空间待探索时都需要解决这一问题。 为应对上述挑战,本文提出了HEHA(基于异构代理的分层探索)方法。该方法利用了一种最近发展的分层方法,将探索过程分解为全局规划和局部规划两个阶段。HEHA的主要贡献在于其全局规划部分,我们提出了一种新的路由算法PEAF(部分任意时间焦点搜索),能够在满足通行性约束的情况下快速找到次优解以最小化代理之间的最大路径长度。 此外,在HEHA的本地规划器中也考虑了异构性的因素,以避免机器人之间重复和冗余的探索。实验结果显示,与基线方法相比,我们的HEHA可以将探索时间减少高达30%。
https://arxiv.org/abs/2510.04161
Traditional object detection models are typically trained on a fixed set of classes, limiting their flexibility and making it costly to incorporate new categories. Open-vocabulary object detection addresses this limitation by enabling models to identify unseen classes without explicit training. Leveraging pretrained models contrastively trained on abundantly available ground-view image-text classification pairs provides a strong foundation for open-vocabulary object detection in aerial imagery. Domain shifts, viewpoint variations, and extreme scale differences make direct knowledge transfer across domains ineffective, requiring specialized adaptation strategies. In this paper, we propose a novel framework for adapting open-vocabulary representations from ground-view images to solve object detection in aerial imagery through structured domain alignment. The method introduces contrastive image-to-image alignment to enhance the similarity between aerial and ground-view embeddings and employs multi-instance vocabulary associations to align aerial images with text embeddings. Extensive experiments on the xView, DOTAv2, VisDrone, DIOR, and HRRSD datasets are used to validate our approach. Our open-vocabulary model achieves improvements of +6.32 mAP on DOTAv2, +4.16 mAP on VisDrone (Images), and +3.46 mAP on HRRSD in the zero-shot setting when compared to finetuned closed-vocabulary dataset-specific model performance, thus paving the way for more flexible and scalable object detection systems in aerial applications.
传统的对象检测模型通常在固定的一组类别上进行训练,这限制了其灵活性,并且增加了将新类别纳入系统的成本。开放词汇对象检测通过使模型能够在没有显式训练的情况下识别未见过的类来解决这一限制。利用预先训练好的模型,这些模型是在大量可用的地平视图像-文本分类对上对比性训练得到的,为在航拍图像中进行开放词汇对象检测提供了坚实的基础。 然而,领域变化、视角差异和极端尺度不同使得直接跨域知识迁移无效,需要专门的适应策略。在这篇论文中,我们提出了一种新颖的框架,用于将地平视图图像中的开放词汇表示适应到解决航拍图像中的对象检测问题,通过结构化的领域对齐来实现这一目标。该方法引入了对比性的图像对齐以增强航拍与地平视图嵌入之间的相似性,并采用多实例词汇关联来使航拍图像与文本嵌入对齐。 我们在xView、DOTAv2、VisDrone、DIOR和HRRSD数据集上进行了广泛的实验,用以验证我们的方法。在零样本设置中,我们的开放词汇模型相比经过微调的封闭词汇特定数据集模型,在DOTAv2上的mAP提升了+6.32%,在VisDrone(图像)上的mAP提升了+4.16%,在HRRSD上提升为+3.46%。这表明了更灵活和可扩展的对象检测系统在航拍应用中的潜力。
https://arxiv.org/abs/2510.03858