We present the first release of SmartWilds, a multimodal wildlife monitoring dataset. SmartWilds is a synchronized collection of drone imagery, camera trap photographs and videos, and bioacoustic recordings collected during summer 2025 at The Wilds safari park in Ohio. This dataset supports multimodal AI research for comprehensive environmental monitoring, addressing critical needs in endangered species research, conservation ecology, and habitat management. Our pilot deployment captured four days of synchronized monitoring across three modalities in a 220-acre pasture containing Pere David's deer, Sichuan takin, Przewalski's horses, as well as species native to Ohio, including bald eagles, white-tailed deer, and coyotes. We provide a comparative analysis of sensor modality performance, demonstrating complementary strengths for landuse patterns, species detection, behavioral analysis, and habitat monitoring. This work establishes reproducible protocols for multimodal wildlife monitoring while contributing open datasets to advance conservation computer vision research. Future releases will include synchronized GPS tracking data from tagged individuals, citizen science data, and expanded temporal coverage across multiple seasons.
我们在此介绍SmartWilds的第一个版本,这是一个多模态野生动物监测数据集。SmartWilds是由无人机影像、相机陷阱拍摄的照片和视频以及生物声学录音组成的一系列同步收集的数据,在2025年夏季于俄亥俄州的The Wilds野生动物园内获取。此数据集支持全面环境监控中的多模式AI研究,满足了濒危物种研究、保护生态学及栖息地管理的关键需求。 我们的试点部署在占地220英亩的牧场上进行了为期四天的跨三个模态同步监测,该牧场包含普尔戴维斯麋鹿、四川羚牛、普氏野马以及俄亥俄州本地物种如白头海雕、白尾鹿和郊狼等。我们提供了传感器模式性能的比较分析,展示了多模态数据在土地使用模式识别、物种检测、行为分析及栖息地监测方面的互补优势。 这项工作建立了可重复使用的多模式野生动物监测协议,并通过提供开放数据集以促进保护计算机视觉研究的发展。未来的数据发布将包括来自标记个体的同步GPS跟踪数据、公民科学数据以及跨越多个季节的时间覆盖扩展。
https://arxiv.org/abs/2509.18894
This article examines state estimation in discrete-time nonlinear stochastic systems with finite-dimensional states and infinite-dimensional measurements, motivated by real-world applications such as vision-based localization and tracking. We develop an extended Kalman filter (EKF) for real-time state estimation, with the measurement noise modeled as an infinite-dimensional random field. When applied to vision-based state estimation, the measurement Jacobians required to implement the EKF are shown to correspond to image gradients. This result provides a novel system-theoretic justification for the use of image gradients as features for vision-based state estimation, contrasting with their (often heuristic) introduction in many computer-vision pipelines. We demonstrate the practical utility of the EKF on a public real-world dataset involving the localization of an aerial drone using video from a downward-facing monocular camera. The EKF is shown to outperform VINS-MONO, an established visual-inertial odometry algorithm, in some cases achieving mean squared error reductions of up to an order of magnitude.
本文探讨了离散时间非线性随机系统中的状态估计问题,这类系统的状态是有限维度的,而测量则是无限维度的。这种研究受到现实世界应用(如基于视觉的定位和跟踪)的启发。我们开发了一种扩展卡尔曼滤波器(EKF),用于实时状态估计,并将测量噪声建模为一个无限维度的随机场。当应用于基于视觉的状态估计时,实施EKF所需的测量雅可比矩阵被证明对应于图像梯度。这一结果提供了一个新颖的系统理论依据,解释了为什么在许多计算机视觉管道中,通常以启发式方式引入的图像梯度可以用作基于视觉状态估计中的特征。 我们通过一个公开的真实世界数据集展示了扩展卡尔曼滤波器的实际效用——该数据集涉及使用向下视图单目摄像头进行空中无人机定位。结果显示,在某些情况下,EKF的表现优于VINS-MONO(一种已经确立的视觉惯性里程计算法),在均方误差方面最多可减少一个数量级。
https://arxiv.org/abs/2509.18749
Formation maintenance with varying number of drones in narrow environments hinders the convergence of planning to the desired configurations. To address this challenge, this paper proposes a formation planning method guided by Deformable Virtual Structures (DVS) with continuous spatiotemporal transformation. Firstly, to satisfy swarm safety distance and preserve formation shape filling integrity for irregular formation geometries, we employ Lloyd algorithm for uniform $\underline{PA}$rtitioning and Hungarian algorithm for $\underline{AS}$signment (PAAS) in DVS. Subsequently, a spatiotemporal trajectory involving DVS is planned using primitive-based path search and nonlinear trajectory optimization. The DVS trajectory achieves adaptive transitions with respect to a varying number of drones while ensuring adaptability to narrow environments through affine transformation. Finally, each agent conducts distributed trajectory planning guided by desired spatiotemporal positions within the DVS, while incorporating collision avoidance and dynamic feasibility requirements. Our method enables up to 15\% of swarm numbers to join or leave in cluttered environments while rapidly restoring the desired formation shape in simulation. Compared to cutting-edge formation planning method, we demonstrate rapid formation recovery capacity and environmental adaptability. Real-world experiments validate the effectiveness and resilience of our formation planning method.
在狭窄环境中,无人机数量变化时的编队维护会妨碍规划向所需配置收敛。为了解决这一挑战,本文提出了一种由可变形虚拟结构(DVS)指导并具备连续时空变换能力的编队规划方法。首先,为了满足群集的安全距离要求,并保持不规则编队几何形状填充完整性的需求,我们使用Lloyd算法进行均匀分区和匈牙利算法进行分配(PAAS)。随后,通过基于基本路径搜索和非线性轨迹优化的方法,在DVS中规划时空轨迹。该DVS轨迹能够随着无人机数量的变化而实现自适应转换,并且通过仿射变换确保了对狭窄环境的适应能力。最后,每个代理在DVS内根据所需的时空位置进行分布式轨迹规划,同时结合碰撞规避和动态可行性要求。 我们的方法使多达15%的群集成员能够在复杂环境中加入或离开,同时能够快速恢复所需的编队形状。与当前最先进的编队规划方法相比,我们展示了快速重组能力和环境适应性。实际实验验证了该编队规划方法的有效性和鲁棒性。
https://arxiv.org/abs/2509.18636
Large vision-language models have driven remarkable progress in open-vocabulary robot policies, e.g., generalist robot manipulation policies, that enable robots to complete complex tasks specified in natural language. Despite these successes, open-vocabulary autonomous drone navigation remains an unsolved challenge due to the scarcity of large-scale demonstrations, real-time control demands of drones for stabilization, and lack of reliable external pose estimation modules. In this work, we present SINGER for language-guided autonomous drone navigation in the open world using only onboard sensing and compute. To train robust, open-vocabulary navigation policies, SINGER leverages three central components: (i) a photorealistic language-embedded flight simulator with minimal sim-to-real gap using Gaussian Splatting for efficient data generation, (ii) an RRT-inspired multi-trajectory generation expert for collision-free navigation demonstrations, and these are used to train (iii) a lightweight end-to-end visuomotor policy for real-time closed-loop control. Through extensive hardware flight experiments, we demonstrate superior zero-shot sim-to-real transfer of our policy to unseen environments and unseen language-conditioned goal objects. When trained on ~700k-1M observation action pairs of language conditioned visuomotor data and deployed on hardware, SINGER outperforms a velocity-controlled semantic guidance baseline by reaching the query 23.33% more on average, and maintains the query in the field of view 16.67% more on average, with 10% fewer collisions.
大型视觉-语言模型在开放式词汇机器人策略方面取得了显著进展,例如通用型机器人操作策略,使机器人能够完成用自然语言指定的复杂任务。尽管取得这些成就,但由于缺乏大规模演示、无人机实时控制的需求以及可靠的外部姿态估计模块不足等原因,开放词汇自主无人机导航仍是一项未解决的挑战。本文介绍了SINGER系统,该系统利用机载传感器和计算能力,在开放世界中实现由语言引导的自主无人机导航。 为了训练稳健的开放式词汇导航策略,SINGER采用了三个核心组件:(i)一个基于高斯点播技术进行高效数据生成、具有最小模拟到真实差距的逼真飞行仿真器;(ii)一种受RRT启发的多轨迹生成专家系统,用于提供无碰撞导航演示。这些被用来训练(iii)一个轻量级端到端视觉运动策略,以实现实时闭环控制。 通过大量的硬件飞行实验,我们证明了我们的政策在未见过的环境和语言条件下的零样本模拟到真实转移具有优越性,并且当在约70万至100万个观察动作对的语言条件下训练后,在部署于实际设备上时,SINGER的表现超过了速度控制的语义引导基线方法:平均而言,前者达到查询目标的概率比后者高出23.33%,保持查询对象在视野中的概率高出了16.67%;并且碰撞次数减少了10%。
https://arxiv.org/abs/2509.18610
The standardisation of Intermodal Loading Units (ILUs), such as containers, semi-trailers and swap bodies, has revolutionised global trade yet their efficient and robust identification remains a critical bottleneck in high-throughput ports and terminals. This paper reviews 63 empirical studies that propose computer vision (CV) based solutions. It covers the last 35 years (1990-2025), tracing the field's evolution from early digital image processing (DIP) and traditional machine learning (ML) to the current dominance of deep learning (DL) techniques. While CV offers cost-effective alternatives for other types of identification techniques, its development is hindered by the lack of publicly available benchmarking datasets. This results in high variance for the reported results such as end-to-end accuracy ranging from 5 % to 96 %. Beyond dataset limitations, this review highlights the emerging challenges especially introduced by the shift from character-based text recognition to scene-text spotting and the integration of mobile cameras (e.g. drones, sensor equipped ground vehicles) for dynamic terminal monitoring. To advance the field, the paper calls for standardised terminology, open-access datasets, shared source code, while outlining future research directions such as contextless text recognition optimised for ISO6346 codes.
集装箱、半拖车和交换车身等国际多式联运装载单元(ILUs)的标准化已经革新了全球贸易,然而,这些装载单元的有效且可靠的识别仍然是高吞吐量港口和码头的关键瓶颈。本文回顾了过去35年(1990-2025)提出的63项基于计算机视觉(CV)解决方案的研究文献,涵盖了从早期数字图像处理(DIP)和传统机器学习(ML)到当前深度学习(DL)技术主导的领域发展进程。 尽管计算机视觉提供了其他识别技术的成本效益替代方案,但其发展受公开可用基准数据集缺乏的阻碍。这导致了报告结果的高度波动性,例如端到端准确性范围从5%至96%不等。除了数据集限制之外,本文还强调了新兴挑战,特别是由基于字符的文字识别转向场景文字检测以及移动摄像头(如无人机、配备传感器的地面车辆)用于动态码头监控带来的挑战。 为了推动该领域的发展,本文呼吁使用标准化术语,开放访问的数据集和共享源代码,并概述未来的研究方向,例如针对ISO6346代码进行优化的无上下文文本识别。
https://arxiv.org/abs/2509.17707
Agile mapless navigation in cluttered 3D environments poses significant challenges for autonomous drones. Conventional mapping-planning-control pipelines incur high computational cost and propagate estimation errors. We present AERO-MPPI, a fully GPU-accelerated framework that unifies perception and planning through an anchor-guided ensemble of Model Predictive Path Integral (MPPI) optimizers. Specifically, we design a multi-resolution LiDAR point-cloud representation that rapidly extracts spatially distributed "anchors" as look-ahead intermediate endpoints, from which we construct polynomial trajectory guides to explore distinct homotopy path classes. At each planning step, we run multiple MPPI instances in parallel and evaluate them with a two-stage multi-objective cost that balances collision avoidance and goal reaching. Implemented entirely with NVIDIA Warp GPU kernels, AERO-MPPI achieves real-time onboard operation and mitigates the local-minima failures of single-MPPI approaches. Extensive simulations in forests, verticals, and inclines demonstrate sustained reliable flight above 7 m/s, with success rates above 80% and smoother trajectories compared to state-of-the-art baselines. Real-world experiments on a LiDAR-equipped quadrotor with NVIDIA Jetson Orin NX 16G confirm that AERO-MPPI runs in real time onboard and consistently achieves safe, agile, and robust flight in complex cluttered environments. The code will be open-sourced upon acceptance of the paper.
在复杂拥挤的3D环境中,自主无人机进行敏捷无地图导航面临重大挑战。传统的方法通常采用映射-规划-控制管道流程,这会带来高昂的计算成本,并且容易传播估计误差。我们提出了一种全新的框架AERO-MPPI(全GPU加速),通过一组由“锚点”引导的模型预测路径积分(MPPI)优化器实现了感知和计划的统一。具体而言,我们设计了一个多分辨率激光雷达点云表示方法,能够快速提取空间分布的“锚点”,作为向前看的中间端点;然后从这些“锚点”构建多项式轨迹指南以探索不同的同伦路径类别。 在每个规划步骤中,AERO-MPPI会并行运行多个MPPI实例,并通过一个两阶段多目标成本函数评估它们的表现,该成本函数平衡了碰撞避免和目标到达之间的权衡。整个框架完全使用NVIDIA Warp GPU内核实现,在飞行器上实现了实时操作,并且缓解了单一MPPI方法容易陷入局部最小值的问题。 在森林、垂直结构以及倾斜地形等环境中的广泛仿真测试表明,AERO-MPPI能够在超过7米/秒的速度下保持可靠的飞行性能,成功率超过了80%,并且相比现有的最先进技术基线,其轨迹更加平滑。此外,在配备激光雷达的四旋翼无人机上进行的真实世界实验也证实了AERO-MPPI能够实时运行并持续实现复杂拥挤环境中的安全、敏捷和稳健飞行。 该论文一旦被接受,代码将公开发布。
https://arxiv.org/abs/2509.17340
Oriented object detection for multi-spectral imagery faces significant challenges due to differences both within and between modalities. Although existing methods have improved detection accuracy through complex network architectures, their high computational complexity and memory consumption severely restrict their performance. Motivated by the success of large kernel convolutions in remote sensing, we propose MO R-CNN, a lightweight framework for multi-spectral oriented detection featuring heterogeneous feature extraction network (HFEN), single modality supervision (SMS), and condition-based multimodal label fusion (CMLF). HFEN leverages inter-modal differences to adaptively align, merge, and enhance multi-modal features. SMS constrains multi-scale features and enables the model to learn from multiple modalities. CMLF fuses multimodal labels based on specific rules, providing the model with a more robust and consistent supervisory signal. Experiments on the DroneVehicle, VEDAI and OGSOD datasets prove the superiority of our method. The source code is available at:this https URL.
面向对象的多光谱图像检测面临由于模态内部和模态之间差异带来的重大挑战。尽管现有的方法通过复杂的网络架构提高了检测精度,但其计算复杂度高、内存消耗大,严重限制了性能的发挥。鉴于在遥感领域大型卷积核的成功应用,我们提出了MO R-CNN,一个轻量级框架用于多光谱定向检测,该框架包括异构特征提取网络(HFEN)、单模态监督(SMS)和基于条件的多模态标签融合(CMLF)。HFEN利用跨模态差异来自适应地对齐、合并并增强多模态特征。SMS限制了多尺度特征,并使模型能够从多种模态中学习。CMLF根据特定规则融合多模态标签,为模型提供了更稳健和一致的监督信号。在DroneVehicle、VEDAI 和 OGSOD 数据集上的实验证明了我们方法的优势。源代码可在以下网址获取:[这个链接](this https URL)。
https://arxiv.org/abs/2509.16957
This paper proposes DroFiT (Drone Frequency lightweight Transformer for speech enhancement, a single microphone speech enhancement network for severe drone self-noise. DroFit integrates a frequency-wise Transformer with a full/sub-band hybrid encoder-decoder and a TCN back-end for memory-efficient streaming. A learnable skip-and-gate fusion with a combined spectral-temporal loss further refines reconstruction. The model is trained on VoiceBank-DEMAND mixed with recorded drone noise (-5 to -25 dB SNR) and evaluate using standard speech enhancement metrics and computational efficiency. Experimental results show that DroFiT achieves competitive enhancement performance while significantly reducing computational and memory demands, paving the way for real-time processing on resource-constrained UAV platforms. Audio demo samples are available on our demo page.
本文提出了DroFiT(Drone Frequency lightweight Transformer,用于语音增强的单麦克风严重无人机自身噪声处理网络)。DroFit 集成了频率感知的Transformer、全频段/子带混合编解码器以及TCN后端,以实现内存高效的流式处理。通过学习型跳过与门控融合技术结合频谱-时序损失进一步优化重建效果。该模型在VoiceBank-DEMAND数据集上进行训练(混入录制的无人机噪声,信噪比为-5至-25dB),并使用标准语音增强指标和计算效率进行评估。实验结果表明,DroFiT 在实现竞争性的语音增强性能的同时显著降低了计算需求和内存占用,为资源受限的无人机平台上的实时处理铺平了道路。音频演示样本可在我们的展示页面上获取。
https://arxiv.org/abs/2509.16945
Ensuring both optimality and safety is critical for the real-world deployment of agents, but becomes particularly challenging when the system dynamics are unknown. To address this problem, we introduce a notion of maximum safe dynamics learning via sufficient exploration in the space of safe policies. We propose a $\textit{pessimistically}$ safe framework that $\textit{optimistically}$ explores informative states and, despite not reaching them due to model uncertainty, ensures continuous online learning of dynamics. The framework achieves first-of-its-kind results: learning the dynamics model sufficiently $-$ up to an arbitrary small tolerance (subject to noise) $-$ in a finite time, while ensuring provably safe operation throughout with high probability and without requiring resets. Building on this, we propose an algorithm to maximize rewards while learning the dynamics $\textit{only to the extent needed}$ to achieve close-to-optimal performance. Unlike typical reinforcement learning (RL) methods, our approach operates online in a non-episodic setting and ensures safety throughout the learning process. We demonstrate the effectiveness of our approach in challenging domains such as autonomous car racing and drone navigation under aerodynamic effects $-$ scenarios where safety is critical and accurate modeling is difficult.
确保代理在现实世界中的部署既具有最优性又安全至关重要,但在系统动力学未知的情况下,这一问题变得尤为棘手。为了解决这个问题,我们引入了一种通过充分探索安全策略空间来学习最大安全动态的概念。我们提出了一种“悲观”地保证安全但“乐观”地探索信息丰富状态的框架,在由于模型不确定性未能达到这些状态的情况下,仍能确保持续在线学习系统动力学。该框架实现了前所未有的成果:在有限时间内以任意小的容忍度(受噪声影响)充分学习动态模型,同时在整个过程中几乎可以肯定地保证安全操作,并且无需重置。在此基础上,我们提出了一种算法,在仅需达到接近最优性能所需的程度的情况下进行动态学习,从而最大化奖励。与典型的强化学习(RL)方法不同,我们的方法在非集中式设置中在线运行,并在整个学习过程中确保安全性。 我们在具有挑战性的领域如自动驾驶汽车赛车和受空气动力学影响的无人机导航中展示了我们方法的有效性——这些场景对安全性和精确建模提出了关键要求且难以实现。
https://arxiv.org/abs/2509.16650
Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet, most existing video understanding and highlight detection methods assume access to the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-step reasoning needed for real-time decision-making. We introduce Aha, an autoregressive highlight detection framework that predicts the relevance of each video frame against a task described in natural language. Without accessing future video frames, Aha utilizes a multimodal vision-language model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To enable scalability, we introduce the Dynamic SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation to capture high-level task objectives, enabling effective frame-level rankings for informativeness, relevance, and uncertainty with respect to the natural language task. Aha achieves state-of-the-art (SOTA) performance on highlight detection benchmarks, surpassing even prior offline, full-context approaches and video-language models by +5.9% on TVSum and +8.3% on this http URL in mAP (mean Average Precision). We explore Aha's potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video. Both experiments demonstrate Aha's potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.
实时理解连续视频流对于在高风险环境中运行的智能代理至关重要,包括自动驾驶汽车、监控无人机和灾害响应机器人。然而,大多数现有的视频理解和亮点检测方法都假设在整个推理过程中可以访问整个视频,这使得它们不适合在线或流媒体场景。特别是,当前模型优化离线总结,无法支持实时决策所需的逐步推理过程。我们引入了Aha框架,这是一个自回归的亮点检测框架,该框架预测每个视频帧相对于描述任务的自然语言的相关性。在不访问未来视频帧的情况下,Aha利用多模态视觉-语言模型和轻量级、解耦头进行训练,并使用一个大型、经过策划的人类为中心的视频标签数据集。 为了实现可扩展性,我们引入了动态SinkCache机制,在无限长度的数据流中实现了恒定的内存使用率,而不会在标准基准测试中降低性能。这鼓励隐藏表示捕捉高层次的任务目标,从而能够有效地根据自然语言任务对帧级的相关性和不确定性进行排名。Aha在亮点检测基准上取得了最先进的(SOTA)性能,超过了之前离线和全上下文的方法,并且比视频-语言模型在TVSum上的mAP提高了+5.9%,在this http URL 上提高了+8.3%。 我们探索了Aha作为实时推理模块的潜力,在给定任务导向的自然语言输入以及连续、以机器人为中心的视频的情况下,应用于现实世界的机器人应用。这两个实验都展示了Aha作为下游规划和长期理解的有效性。
https://arxiv.org/abs/2509.16421
We present Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories (ACDC), an autonomous drone cinematography system driven by natural language communication between human directors and drones. The main limitation of previous drone cinematography workflows is that they require manual selection of waypoints and view angles based on predefined human intent, which is labor-intensive and yields inconsistent performance. In this paper, we propose employing large language models (LLMs) and vision foundation models (VFMs) to convert free-form natural language prompts directly into executable indoor UAV video tours. Specifically, our method comprises a vision-language retrieval pipeline for initial waypoint selection, a preference-based Bayesian optimization framework that refines poses using aesthetic feedback, and a motion planner that generates safe quadrotor trajectories. We validate ACDC through both simulation and hardware-in-the-loop experiments, demonstrating that it robustly produces professional-quality footage across diverse indoor scenes without requiring expertise in robotics or cinematography. These results highlight the potential of embodied AI agents to close the loop from open-vocabulary dialogue to real-world autonomous aerial cinematography.
我们介绍了代理式空中摄影系统“从对话提示到电影轨迹”(ACDC),这是一个由人与无人机之间自然语言交流驱动的自主无人机摄影系统。以往无人机摄影工作流程的主要限制在于,它们需要手动选择航点和视角角度,这些选择基于预先定义的人类意图进行,这既费时又会导致性能不一致。在本文中,我们提出使用大型语言模型(LLMs)和视觉基础模型(VFMs),将自由形式的自然语言提示直接转换为可执行的室内无人机视频游览。具体而言,我们的方法包括一个用于初始航点选择的视觉-语言检索管道、一个基于偏好的贝叶斯优化框架,该框架使用美学反馈来精炼姿态,并且还有一个运动规划器生成安全的四旋翼轨迹。 我们通过仿真和硬件在环实验验证了ACDC的有效性,证明它能够稳健地在各种室内场景中生成专业级别的影像素材,而无需具备机器人技术或电影摄影的专业知识。这些结果突显了具身AI代理从开放式词汇对话到现实世界自主空中摄影闭环的潜力。
https://arxiv.org/abs/2509.16176
Drones have recently emerged as a faster, safer, and cost-efficient way for last-mile deliveries of parcels, particularly for urgent medical deliveries highlighted during the pandemic. This paper addresses a new challenge of multi-parcel delivery with a swarm of energy-aware drones, accounting for time-sensitive customer requirements. Each drone plans an optimal multi-parcel route within its battery-restricted flight range to minimize delivery delays and reduce energy consumption. The problem is tackled by decomposing it into three sub-problems: (1) optimizing depot locations and service areas using K-means clustering; (2) determining the optimal flight range for drones through reinforcement learning; and (3) planning and selecting multi-parcel delivery routes via a new optimized plan selection approach. To integrate these solutions and enhance long-term efficiency, we propose a novel algorithm leveraging actor-critic-based multi-agent deep reinforcement learning. Extensive experimentation using realistic delivery datasets demonstrate an exceptional performance of the proposed algorithm. We provide new insights into economic efficiency (minimize energy consumption), rapid operations (reduce delivery delays and overall execution time), and strategic guidance on depot deployment for practical logistics applications.
最近,无人机因其速度快、安全性高以及成本效益好而成为最后一公里包裹递送(特别是紧急医疗递送)的新选择,在疫情期间尤为突出。本文研究了一个新的挑战:如何利用一群节能意识的无人机进行多件包裹配送,并满足客户的时间敏感需求。每个无人机在电池限制的飞行范围内计划最优的多件包裹路线,以减少交付延迟并降低能耗。为解决这一问题,我们将它分解成三个子问题: 1. 使用K-means聚类优化集散中心的位置和服务区域; 2. 通过强化学习确定无人机的最佳飞行范围; 3. 采用新的最佳方案选择方法规划和选择多件包裹的递送路线。 为了整合这些解决方案并提升长期效率,我们提出了一种基于演员-评论家模型的多代理深度强化学习的新算法。使用实际配送数据集进行广泛的实验表明了所提算法的卓越性能。本文提供了关于经济效率(最小化能耗)、快速操作(减少交付延迟和总体执行时间)以及集散中心部署的战略指导,为实际物流应用提供新的见解。
https://arxiv.org/abs/2509.15830
In the realm of fully autonomous drones, the accurate detection of obstacles is paramount to ensure safe navigation and prevent collisions. Among these challenges, the detection of wires stands out due to their slender profile, which poses a unique and intricate problem. To address this issue, we present an innovative solution in the form of a monocular end-to-end model for wire segmentation and depth estimation. Our approach leverages a temporal correlation layer trained on synthetic data, providing the model with the ability to effectively tackle the complex joint task of wire detection and depth estimation. We demonstrate the superiority of our proposed method over existing competitive approaches in the joint task of wire detection and depth estimation. Our results underscore the potential of our model to enhance the safety and precision of autonomous drones, shedding light on its promising applications in real-world scenarios.
在完全自主无人机领域,准确检测障碍物是确保安全导航和防止碰撞的关键。其中,电线的检测尤为突出,因为它们纤细的外形构成了一个独特而复杂的难题。为了解决这个问题,我们提出了一种创新的解决方案——一种单目端到端模型,用于电线分割和深度估计。我们的方法利用了在合成数据上训练的时间相关层,赋予模型有效处理电线检测和深度估计这一复杂联合任务的能力。我们在电线检测与深度估计的联合任务中展示了我们提出的方案优于现有的竞争性方法。我们的结果强调了该模型提升自主无人机安全性和精度的巨大潜力,并揭示了它在实际应用中的广阔前景。
https://arxiv.org/abs/2509.14989
While model-based controllers have demonstrated remarkable performance in autonomous drone racing, their performance is often constrained by the reliance on pre-computed reference trajectories. Conventional approaches, such as trajectory tracking, demand a dynamically feasible, full-state reference, whereas contouring control relaxes this requirement to a geometric path but still necessitates a reference. Recent advancements in reinforcement learning (RL) have revealed that many model-based controllers optimize surrogate objectives, such as trajectory tracking, rather than the primary racing goal of directly maximizing progress through gates. Inspired by these findings, this work introduces a reference-free method for time-optimal racing by incorporating this gate progress objective, derived from RL reward shaping, directly into the Model Predictive Path Integral (MPPI) formulation. The sampling-based nature of MPPI makes it uniquely capable of optimizing the discontinuous and non-differentiable objective in real-time. We also establish a unified framework that leverages MPPI to systematically and fairly compare three distinct objective functions with a consistent dynamics model and parameter set: classical trajectory tracking, contouring control, and the proposed gate progress objective. We compare the performance of these three objectives when solved via both MPPI and a traditional gradient-based solver. Our results demonstrate that the proposed reference-free approach achieves competitive racing performance, rivaling or exceeding reference-based methods. Videos are available at this https URL
虽然基于模型的控制器在自主无人机竞速中展现了卓越的表现,但它们的性能往往受限于对预计算参考轨迹的依赖。传统的路径跟踪方法需要动态可行且完整的状态参考,而轮廓控制则放松了这一要求,仅需几何路径作为参考,但仍存在依赖参考的问题。近期强化学习(RL)的研究表明,许多基于模型的控制器优化的是替代目标函数,例如路径跟踪,而非直接最大化通过门的速度这一主要竞速目标。受此启发,本文提出了一种无参考的方法,用于时间最优竞速,在Model Predictive Path Integral (MPPI) 方法中直接应用了从RL奖励塑形得出的门进度目标。由于MPPI方法基于采样特性,使其能够实时优化这一不连续且非可微的目标函数。此外,我们还建立了一个统一框架,利用MPPI系统地和公平地比较三种不同的目标函数,并采用一致的动力学模型和参数集:经典路径跟踪、轮廓控制以及所提出的门进度目标。在使用MPPI和传统梯度法求解时,我们评估了这三项目标函数的性能。实验结果表明,所提出的无参考方法达到了与基于参考的方法相当甚至更优的竞争性竞速表现。相关视频可在以下链接查看:[https URL]
https://arxiv.org/abs/2509.14726
We present a cooperative aerial-ground search-and-rescue (SAR) framework that pairs two unmanned aerial vehicles (UAVs) with an unmanned ground vehicle (UGV) to achieve rapid victim localization and obstacle-aware navigation in unknown environments. We dub this framework Guided Long-horizon Integrated Drone Escort (GLIDE), highlighting the UGV's reliance on UAV guidance for long-horizon planning. In our framework, a goal-searching UAV executes real-time onboard victim detection and georeferencing to nominate goals for the ground platform, while a terrain-scouting UAV flies ahead of the UGV's planned route to provide mid-level traversability updates. The UGV fuses aerial cues with local sensing to perform time-efficient A* planning and continuous replanning as information arrives. Additionally, we present a hardware demonstration (using a GEM e6 golf cart as the UGV and two X500 UAVs) to evaluate end-to-end SAR mission performance and include simulation ablations to assess the planning stack in isolation from detection. Empirical results demonstrate that explicit role separation across UAVs, coupled with terrain scouting and guided planning, improves reach time and navigation safety in time-critical SAR missions.
我们提出了一种合作的空地搜救(SAR)框架,该框架将两架无人驾驶飞机(UAV)与一辆无人驾驶地面车辆(UGV)配对,以实现快速定位受害者和感知障碍物进行导航的目标,在未知环境中完成任务。我们将这个框架命名为“Guided Long-horizon Integrated Drone Escort”(GLIDE),强调了UGV依赖于无人机的引导来进行长期规划的特点。 在我们的框架中,一架执行目标搜索任务的UAV实时地进行机载受害者检测和地理参考定位,并为地面平台提名目标位置。另一架地形侦察的UAV则飞越UGV预定路线的前方,提供中间层级的可通行性更新信息。UGV融合来自空中的线索与本地感应信息来执行高效的A*路径规划,并在信息不断到来时进行持续的重新规划。 此外,我们还通过硬件演示(使用GEM e6高尔夫球车作为UGV和两个X500 UAV)评估了端到端搜救任务性能,并提供了模拟消融研究以独立于检测功能来评估规划栈的效果。实证结果表明,在时间紧迫的情况下进行搜救行动时,明确划分UAV的角色分工,结合地形侦察与引导式规划能够提高到达时间和导航安全性。
https://arxiv.org/abs/2509.14210
We present FlightDiffusion, a diffusion-model-based framework for training autonomous drones from first-person view (FPV) video. Our model generates realistic video sequences from a single frame, enriched with corresponding action spaces to enable reasoning-driven navigation in dynamic environments. Beyond direct policy learning, FlightDiffusion leverages its generative capabilities to synthesize diverse FPV trajectories and state-action pairs, facilitating the creation of large-scale training datasets without the high cost of real-world data collection. Our evaluation demonstrates that the generated trajectories are physically plausible and executable, with a mean position error of 0.25 m (RMSE 0.28 m) and a mean orientation error of 0.19 rad (RMSE 0.24 rad). This approach enables improved policy learning and dataset scalability, leading to superior performance in downstream navigation tasks. Results in simulated environments highlight enhanced robustness, smoother trajectory planning, and adaptability to unseen conditions. An ANOVA revealed no statistically significant difference between performance in simulation and reality (F(1, 16) = 0.394, p = 0.541), with success rates of M = 0.628 (SD = 0.162) and M = 0.617 (SD = 0.177), respectively, indicating strong sim-to-real transfer. The generated datasets provide a valuable resource for future UAV research. This work introduces diffusion-based reasoning as a promising paradigm for unifying navigation, action generation, and data synthesis in aerial robotics.
我们介绍了FlightDiffusion,这是一个基于扩散模型的框架,用于从第一人称视角(FPV)视频训练自主无人机。我们的模型可以从单帧生成逼真的视频序列,并添加相应的行为空间以在动态环境中实现推理驱动导航。除了直接的学习策略之外,FlightDiffusion 还利用其生成能力来合成多样化的 FPV 轨迹和状态-动作对,从而无需高昂的实际数据收集成本即可创建大规模训练数据集。 我们的评估表明,生成的轨迹具有物理上的合理性及可执行性,平均位置误差为 0.25 米(均方根误差 RMSE:0.28 米),平均方向误差为 0.19 弧度(RMSE:0.24 弧度)。这种方法能够提升策略学习和数据集规模的扩展能力,在下游导航任务中表现出色。模拟环境中的结果表明了增强的鲁棒性、更加平滑的轨迹规划以及对未见条件的适应能力。 方差分析揭示出,仿真与现实性能之间没有显著统计差异(F(1, 16) = 0.394,p = 0.541),成功率分别为 M = 0.628 (SD = 0.162) 和 M = 0.617 (SD = 0.177),表明了强大的仿真到现实的转换能力。生成的数据集为未来的无人机研究提供了宝贵的资源。 这项工作引入基于扩散模型的推理作为统一空中机器人导航、动作产生和数据合成的有前景范式。
https://arxiv.org/abs/2509.14082
Modular Aerial Robot Systems (MARS) consist of multiple drone modules that are physically bound together to form a single structure for flight. Exploiting structural redundancy, MARS can be reconfigured into different formations to mitigate unit or rotor failures and maintain stable flight. Prior work on MARS self-reconfiguration has solely focused on maximizing controllability margins to tolerate a single rotor or unit fault for rectangular-shaped MARS. We propose TransforMARS, a general fault-tolerant reconfiguration framework that transforms arbitrarily shaped MARS under multiple rotor and unit faults while ensuring continuous in-air stability. Specifically, we develop algorithms to first identify and construct minimum controllable assemblies containing faulty units. We then plan feasible disassembly-assembly sequences to transport MARS units or subassemblies to form target configuration. Our approach enables more flexible and practical feasible reconfiguration. We validate TransforMARS in challenging arbitrarily shaped MARS configurations, demonstrating substantial improvements over prior works in both the capacity of handling diverse configurations and the number of faults tolerated. The videos and source code of this work are available at the anonymous repository: this https URL
模块化空中机器人系统(Modular Aerial Robot Systems,简称MARS)由多个物理连接在一起的无人机模块组成,形成一个单一结构进行飞行。通过利用结构冗余性,MARS可以重新配置成不同的形式来缓解单元或旋翼故障,并保持稳定的飞行状态。先前关于MARS自重构的研究主要集中在最大化控制裕度以容忍矩形形状的MARS中的单个旋翼或单元故障。 我们提出了TransforMARS,这是一个通用的容错重配置框架,它可以在面对多个旋翼和单元故障的情况下对任意形状的MARS进行变换,并确保连续的空中稳定性。具体来说,我们开发了算法来首先识别并构建包含故障单元的最小可控组件。然后规划可行的拆装序列,将MARS单元或子组件移动到目标配置中。我们的方法使更灵活和实用的重配置成为可能。 我们在具有挑战性的任意形状MARS配置中验证了TransforMARS的有效性,并在处理多样化配置的能力以及容忍故障的数量方面都显著超过了先前的工作成果。这项工作的视频和源代码可以在匿名仓库中找到:[提供的URL链接]。
https://arxiv.org/abs/2509.14025
Drone detection in visually complex environments remains challenging due to background clutter, small object scale, and camouflage effects. While generic object detectors like YOLO exhibit strong performance in low-texture scenes, their effectiveness degrades in cluttered environments with low object-background separability. To address these limitations, this work presents an enhanced iteration of YOLO-FEDER FusionNet -- a detection framework that integrates generic object detection with camouflage object detection techniques. Building upon the original architecture, the proposed iteration introduces systematic advancements in training data composition, feature fusion strategies, and backbone design. Specifically, the training process leverages large-scale, photo-realistic synthetic data, complemented by a small set of real-world samples, to enhance robustness under visually complex conditions. The contribution of intermediate multi-scale FEDER features is systematically evaluated, and detection performance is comprehensively benchmarked across multiple YOLO-based backbone configurations. Empirical results indicate that integrating intermediate FEDER features, in combination with backbone upgrades, contributes to notable performance improvements. In the most promising configuration -- YOLO-FEDER FusionNet with a YOLOv8l backbone and FEDER features derived from the DWD module -- these enhancements lead to a FNR reduction of up to 39.1 percentage points and a mAP increase of up to 62.8 percentage points at an IoU threshold of 0.5, compared to the initial baseline.
在复杂视觉环境中检测无人机仍面临挑战,主要是由于背景杂乱、小型物体尺度以及伪装效果。尽管像YOLO这样的通用对象检测器在低纹理场景中表现出色,但在背景与目标分离度较低的混乱环境中,其性能会大幅下降。为了解决这些问题,本研究提出了一种增强版的YOLO-FEDER FusionNet——一种结合了通用物体检测和伪装物体检测技术的检测框架。基于原有架构,这项工作在训练数据组成、特征融合策略以及骨干网络设计方面进行了系统性改进。 具体来说,在训练过程中,该方法利用大规模逼真的合成数据,并辅以少量真实世界样本,以此增强其在复杂视觉环境中的鲁棒性。同时,对中间多尺度FEDER特征的贡献进行了一系列表征分析,并针对多个基于YOLO的骨干网络配置进行了全面的检测性能基准测试。 实证结果显示,在结合中间层FEDER特征和升级后的骨干网络的情况下,性能得到了显著提升。在最具有前景的配置——即采用YOLOv8l骨干网和来自DWD模块衍生出的FEDER特征的YOLO-FEDER FusionNet中——与初始基准相比,在IoU阈值为0.5时,实现了高达39.1个百分点的漏报率(FNR)降低以及最高62.8个百分点的平均精度(mAP)提升。
https://arxiv.org/abs/2509.14012
Unmanned Aerial Vehicles (UAVs) are increasingly used in automated inspection, delivery, and navigation tasks that require reliable autonomy. This project develops a reinforcement learning (RL) approach to enable a single UAV to autonomously navigate between predefined points without manual intervention. The drone learns navigation policies through trial-and-error interaction, using a custom reward function that encourages goal-reaching efficiency while penalizing collisions and unsafe behavior. The control system integrates ROS with a Gym-compatible training environment, enabling flexible deployment and testing. After training, the learned policy is deployed on a real UAV platform and evaluated under practical conditions. Results show that the UAV can successfully perform autonomous navigation with minimal human oversight, demonstrating the viability of RL-based control for point-to-point drone operations in real-world scenarios.
无人飞行器(UAV)在自动化检查、交付和导航任务中越来越常用,这些任务需要可靠的自主性。本项目开发了一种强化学习(RL)方法,使单个无人机能够在预定义的点之间自主导航,并且无需人工干预。通过试错互动,无人机使用自定义奖励函数来学习导航策略,该奖励函数鼓励达到目标的有效性并惩罚碰撞和不安全行为。控制系统结合了ROS与兼容Gym的训练环境,允许灵活部署和测试。经过训练后,所学政策被部署到真实的UAV平台上,并在实际条件下进行评估。 结果显示,无人机能够在最少的人类监督下成功执行自主导航任务,这表明基于强化学习控制的方法对于现实世界中点对点无人机操作具有可行性。
https://arxiv.org/abs/2509.13943
This paper presents a novel three-stage framework for real-time foreign object intrusion (FOI) detection and tracking in power transmission systems. The framework integrates: (1) a YOLOv7 segmentation model for fast and robust object localization, (2) a ConvNeXt-based feature extractor trained with triplet loss to generate discriminative embeddings, and (3) a feature-assisted IoU tracker that ensures resilient multi-object tracking under occlusion and motion. To enable scalable field deployment, the pipeline is optimized for deployment on low-cost edge hardware using mixed-precision inference. The system supports incremental updates by adding embeddings from previously unseen objects into a reference database without requiring model retraining. Extensive experiments on real-world surveillance and drone video datasets demonstrate the framework's high accuracy and robustness across diverse FOI scenarios. In addition, hardware benchmarks on NVIDIA Jetson devices confirm the framework's practicality and scalability for real-world edge applications.
本文提出了一种新颖的三阶段框架,用于电力传输系统中实时异物入侵(FOI)检测和跟踪。该框架整合了以下三个部分: 1. 一个基于YOLOv7的分割模型,用于快速且稳健的对象定位; 2. 基于ConvNeXt的特征提取器,并使用三元组损失进行训练,以生成具有区分性的嵌入向量; 3. 一种辅助特征的IoU跟踪器,在遮挡和运动情况下确保多对象跟踪的鲁棒性。 为了实现可扩展的实际部署,该流程被优化为可以在低成本边缘硬件上使用混合精度推理进行部署。系统支持增量更新,通过将来自之前未见过的对象的嵌入添加到参考数据库中来完成,而无需重新训练模型。在现实世界的监控和无人机视频数据集上的广泛实验表明,该框架在各种FOI场景下具有高准确性和鲁棒性。 此外,在NVIDIA Jetson设备上进行的硬件基准测试确认了该框架的实际可行性和在边缘应用中的可扩展性。
https://arxiv.org/abs/2509.13396