Automating drone-assisted processes is a complex task. Many solutions rely on trajectory generation and tracking, whereas in contrast, path-following control is a particularly promising approach, offering an intuitive and natural approach to automate tasks for drones and other vehicles. While different solutions to the path-following problem have been proposed, most of them lack the capability to explicitly handle state and input constraints, are formulated in a conservative two-stage approach, or are only applicable to linear systems. To address these challenges, the paper is built upon a Model Predictive Control-based path-following framework and extends its application to the Crazyflie quadrotor, which is investigated in hardware experiments. A cascaded control structure including an underlying attitude controller is included in the Model Predictive Path-Following Control formulation to meet the challenging real-time demands of quadrotor control. The effectiveness of the proposed method is demonstrated through real-world experiments, representing, to the best of the authors' knowledge, a novel application of this MPC-based path-following approach to the quadrotor. Additionally, as an extension to the original method, to allow for deviations of the path in cases where the precise following of the path might be overly restrictive, a corridor path-following approach is presented.
自动化无人机辅助流程是一项复杂的任务。许多解决方案依赖于轨迹生成和跟踪,相比之下,路径跟随控制是一种特别有前景的方法,它提供了一种直观且自然的方式来为无人机和其他车辆实现任务的自动化。尽管已经提出了多种解决路径跟随问题的不同方案,但大多数缺乏明确处理状态和输入约束的能力、采用保守的两阶段方法,或者仅适用于线性系统。为了应对这些挑战,该论文基于模型预测控制(Model Predictive Control, MPC)的路径跟随框架,并将其应用于Crazyflie四旋翼机,在硬件实验中进行了研究。在模型预测路径跟随控制公式中,包括了包含底层姿态控制器的级联控制系统,以满足四旋翼机实时控制的挑战性需求。通过真实世界实验展示了所提方法的有效性,据作者所知,这是将基于MPC的路径跟随方法应用于四旋翼的新颖应用。此外,作为对原始方法的扩展,在精确遵循路径可能过于严格的场合中,提出了一种走廊路径跟随的方法,允许路径有一定的偏离。
https://arxiv.org/abs/2506.15447
Autonomous navigation by drones using onboard sensors combined with machine learning and computer vision algorithms is impacting a number of domains, including agriculture, logistics, and disaster management. In this paper, we examine the use of drones for assisting visually impaired people (VIPs) in navigating through outdoor urban environments. Specifically, we present a perception-based path planning system for local planning around the neighborhood of the VIP, integrated with a global planner based on GPS and maps for coarse planning. We represent the problem using a geometric formulation and propose a multi DNN based framework for obstacle avoidance of the UAV as well as the VIP. Our evaluations conducted on a drone human system in a university campus environment verifies the feasibility of our algorithms in three scenarios; when the VIP walks on a footpath, near parked vehicles, and in a crowded street.
使用机载传感器结合机器学习和计算机视觉算法的自主无人机导航正在影响包括农业、物流和灾害管理在内的多个领域。在本文中,我们探讨了利用无人机协助视障人士(VIP)在户外城市环境中导航的应用。具体而言,我们提出了一种基于感知的地方路径规划系统,用于在VIP周围区域进行本地规划,并结合基于GPS和地图的全局规划器来进行粗略规划。我们采用几何方法来表述这个问题,并提出了一个多深度神经网络(DNN)框架,以实现无人机及其用户的避障功能。 我们在大学校园环境中对无人机与人组成的系统进行了评估,在三种场景中验证了我们的算法的有效性:当VIP在人行道上行走、靠近停放的车辆以及在繁忙街道上行走时。
https://arxiv.org/abs/2506.14857
Autonomous drones capable of interpreting and executing high-level language instructions in unstructured environments remain a long-standing goal. Yet existing approaches are constrained by their dependence on hand-crafted skills, extensive parameter tuning, or computationally intensive models unsuitable for onboard use. We introduce GRaD-Nav++, a lightweight Vision-Language-Action (VLA) framework that runs fully onboard and follows natural-language commands in real time. Our policy is trained in a photorealistic 3D Gaussian Splatting (3DGS) simulator via Differentiable Reinforcement Learning (DiffRL), enabling efficient learning of low-level control from visual and linguistic inputs. At its core is a Mixture-of-Experts (MoE) action head, which adaptively routes computation to improve generalization while mitigating forgetting. In multi-task generalization experiments, GRaD-Nav++ achieves a success rate of 83% on trained tasks and 75% on unseen tasks in simulation. When deployed on real hardware, it attains 67% success on trained tasks and 50% on unseen ones. In multi-environment adaptation experiments, GRaD-Nav++ achieves an average success rate of 81% across diverse simulated environments and 67% across varied real-world settings. These results establish a new benchmark for fully onboard Vision-Language-Action (VLA) flight and demonstrate that compact, efficient models can enable reliable, language-guided navigation without relying on external infrastructure.
自主无人机能够解读并执行在无结构环境中下达的高级语言指令,一直是长久以来的目标。然而,现有的方法受限于手工设计技能、大量参数调整或计算密集型模型的使用,后者不适合机载部署。我们引入了GRaD-Nav++,这是一个轻量级的视觉-语言-行动(VLA)框架,能够在无人机上完全独立运行,并实时遵循自然语言指令。 我们的策略通过可微强化学习(DiffRL)在逼真的3D高斯点集(3DGS)模拟器中进行训练,从视觉和语言输入中高效地学习低级控制。核心部分是一个专家混合模型(MoE),该模型能够自适应地分配计算资源以提高泛化能力,并减少遗忘效应。 在多任务泛化实验中,GRaD-Nav++在仿真环境中针对已训练的任务达到了83%的成功率,对于未见过的任务为75%。当部署到真实硬件上时,它对已训练的任务的成功率为67%,而面对未知环境下的任务成功率则为50%。 在多环境适应实验中,GRaD-Nav++在多样化的仿真环境中平均成功率达到81%,而在不同的现实世界设置下则达到了67%。这些结果为完全机载的视觉语言行动飞行设定了新的基准,并证明了紧凑且高效的模型能够在无需依赖外部基础设施的情况下实现可靠的语言引导导航。
https://arxiv.org/abs/2506.14009
Commercial UAVs are an emerging security threat as they are capable of carrying hazardous payloads or disrupting air traffic. To counter UAVs, we introduce an autonomous 3D target encirclement and interception strategy. Unlike traditional ground-guided systems, this strategy employs autonomous drones to track and engage non-cooperative hostile UAVs, which is effective in non-line-of-sight conditions, GPS denial, and radar jamming, where conventional detection and neutralization from ground guidance fail. Using two noisy real-time distances measured by drones, guardian drones estimate the relative position from their own to the target using observation and velocity compensation methods, based on anti-synchronization (AS) and an X$-$Y circular motion combined with vertical jitter. An encirclement control mechanism is proposed to enable UAVs to adaptively transition from encircling and protecting a target to encircling and monitoring a hostile target. Upon breaching a warning threshold, the UAVs may even employ a suicide attack to neutralize the hostile target. We validate this strategy through real-world UAV experiments and simulated analysis in MATLAB, demonstrating its effectiveness in detecting, encircling, and intercepting hostile drones. More details: this https URL.
商用无人机因其能够携带危险载荷或干扰空中交通,正成为一个新兴的安全威胁。为了应对这一挑战,我们提出了一种自主的3D目标包围和拦截策略。与传统的地面引导系统不同,该策略采用自主无人机来追踪并打击非合作敌对无人机,在视线受阻、GPS信号丢失及雷达干扰等情况下尤为有效,这些都是传统从地面指挥进行检测和中立化措施失效的情况。 通过使用两台无人机测量的实时噪声距离数据,守护无人机能够根据反同步(AS)机制以及结合X-Y圆周运动与垂直抖动的方法来估算自身到目标的相对位置,并运用观测和速度补偿技术。我们还提出了一种包围控制机制,使无人机能够在保护或监视敌对目标时灵活转变策略。 一旦越过预警阈值,无人机甚至可能采取自杀式攻击方式以中立化敌对目标。我们在现实世界的无人机实验以及MATLAB中的仿真分析验证了这一策略的有效性,证明其在检测、包围和拦截敌对无人机方面的能力显著增强。 更多详情,请参阅:[此处插入链接]
https://arxiv.org/abs/2506.13106
Small object detection in UAV imagery is crucial for applications such as search-and-rescue, traffic monitoring, and environmental surveillance, but it is hampered by tiny object size, low signal-to-noise ratios, and limited feature extraction. Existing multi-scale fusion methods help, but add computational burden and blur fine details, making small object detection in cluttered scenes difficult. To overcome these challenges, we propose the Multi-scale Global-detail Feature Integration Strategy (MGDFIS), a unified fusion framework that tightly couples global context with local detail to boost detection performance while maintaining efficiency. MGDFIS comprises three synergistic modules: the FusionLock-TSS Attention Module, which marries token-statistics self-attention with DynamicTanh normalization to highlight spectral and spatial cues at minimal cost; the Global-detail Integration Module, which fuses multi-scale context via directional convolution and parallel attention while preserving subtle shape and texture variations; and the Dynamic Pixel Attention Module, which generates pixel-wise weighting maps to rebalance uneven foreground and background distributions and sharpen responses to true object regions. Extensive experiments on the VisDrone benchmark demonstrate that MGDFIS consistently outperforms state-of-the-art methods across diverse backbone architectures and detection frameworks, achieving superior precision and recall with low inference time. By striking an optimal balance between accuracy and resource usage, MGDFIS provides a practical solution for small-object detection on resource-constrained UAV platforms.
在无人机影像中进行小目标检测对于搜索与救援、交通监控和环境监视等应用至关重要,但该任务受到小型目标尺寸微小、信噪比低以及特征提取有限等因素的限制。现有的多尺度融合方法虽然有所帮助,但也增加了计算负担并模糊了细节点,使得在复杂场景中小目标的检测变得困难。 为克服这些挑战,我们提出了一个多尺度全局-细节特性整合策略(MGDFIS),这是一种统一的融合框架,将全球上下文与局部详情紧密地结合在一起,以提升检测性能同时保持效率。MGDFIS包括三个协同模块:FusionLock-TSS注意模块通过将令牌统计自注意力与DynamicTanh归一化相结合,在最小成本下突出显示光谱和空间线索;全局-细节整合模块通过定向卷积和平行注意方式融合多尺度上下文,同时保留细微的形状和纹理变化;动态像素注意模块生成逐像素加权图以重新平衡前景和背景分布,并使对真实目标区域的响应更加清晰。 在VisDrone基准测试中的大量实验表明,MGDFIS在整个骨干架构和检测框架中始终优于最先进的方法,在精度和召回率方面表现出色且推理时间短。通过在准确性与资源使用之间找到最佳平衡点,MGDFIS为资源受限的无人机平台上的小目标检测提供了一种实用解决方案。
https://arxiv.org/abs/2506.12697
Accurate rotational odometry is crucial for autonomous robotic systems, particularly for small, power-constrained platforms such as drones and mobile robots. This study introduces thermal-gyro fusion, a novel sensor fusion approach that integrates ultra-low-resolution thermal imaging with gyroscope readings for rotational odometry. Unlike RGB cameras, thermal imaging is invariant to lighting conditions and, when fused with gyroscopic data, mitigates drift which is a common limitation of inertial sensors. We first develop a multimodal data acquisition system to collect synchronized thermal and gyroscope data, along with rotational speed labels, across diverse environments. Subsequently, we design and train a lightweight Convolutional Neural Network (CNN) that fuses both modalities for rotational speed estimation. Our analysis demonstrates that thermal-gyro fusion enables a significant reduction in thermal camera resolution without significantly compromising accuracy, thereby improving computational efficiency and memory utilization. These advantages make our approach well-suited for real-time deployment in resource-constrained robotic systems. Finally, to facilitate further research, we publicly release our dataset as supplementary material.
精确的旋转里程计对于自主机器人系统至关重要,尤其是在小型、受功率限制的平台(如无人机和移动机器人)上。本研究介绍了热像仪-陀螺仪融合(thermal-gyro fusion),这是一种新颖的传感器融合方法,它将超低分辨率热成像与陀螺仪读数相结合用于旋转里程计。不同于RGB相机,热成像不受照明条件的影响,并且当与陀螺数据结合使用时,可以缓解惯性传感器常见的漂移问题。 我们首先开发了一个多模态数据采集系统,用于收集同步的热像和陀螺仪数据以及不同环境中的旋转速度标签。随后,设计并训练了一种轻量级卷积神经网络(CNN),该网络融合这两种模式以进行旋转速度估计。我们的分析表明,通过热像-陀螺仪融合可以在不显著影响精度的情况下大幅降低热成像相机的分辨率,从而提高计算效率和内存利用率。这些优点使我们的方法非常适合在资源受限的机器人系统中实时部署。 最后,为了促进进一步的研究,我们将数据集作为补充材料公开发布。
https://arxiv.org/abs/2506.12536
Autonomous drones are rapidly reshaping industries ranging from aerial delivery and infrastructure inspection to environmental monitoring and disaster response. Ensuring the safety, reliability, and efficiency of these systems is paramount as they transition from research prototypes to mission-critical platforms. This paper presents a step-by-step guide to establishing a robust autonomous drone testing pipeline, covering each critical stage: Software-in-the-Loop (SIL) Simulation Testing, Hardware-in-the-Loop (HIL) Testing, Controlled Real-World Testing, and In-Field Testing. Using practical examples, including the marker-based autonomous landing system, we demonstrate how to systematically verify drone system behaviors, identify integration issues, and optimize performance. Furthermore, we highlight emerging trends shaping the future of drone testing, including the integration of Neurosymbolic and LLMs, creating co-simulation environments, and Digital Twin-enabled simulation-based testing techniques. By following this pipeline, developers and researchers can achieve comprehensive validation, minimize deployment risks, and prepare autonomous drones for safe and reliable real-world operations.
自主无人机正在迅速重塑从空中交付和基础设施检查到环境监测和灾害响应等多个行业的格局。随着这些系统从研究原型转变为关键任务平台,确保它们的安全性、可靠性和效率至关重要。本文提供了一步步建立强大的自主无人机测试流水线的指南,涵盖了每一个关键阶段:软件在环(SIL)仿真测试、硬件在环(HIL)测试、受控现实世界测试以及现场测试。通过包括基于标记的自主着陆系统的实际示例,我们展示了如何系统地验证无人机系统的行为、识别集成问题并优化性能。此外,本文还强调了塑造未来无人机测试趋势的新兴技术,如神经符号和LLM(大型语言模型)的整合、协同仿真环境的创建以及数字孪生支持的基于仿真的测试技术。通过遵循这一流水线,开发人员和研究人员可以实现全面验证,最小化部署风险,并为自主无人机在现实世界的可靠运行做好准备。
https://arxiv.org/abs/2506.11400
The increase in vehicle numbers in California, driven by inadequate transportation systems and sparse speed cameras, necessitates effective vehicle speed detection. Detecting vehicle speeds per lane is critical for monitoring High-Occupancy Vehicle (HOV) lane speeds, distinguishing between cars and heavy vehicles with differing speed limits, and enforcing lane restrictions for heavy vehicles. While prior works utilized YOLO (You Only Look Once) for vehicle speed detection, they often lacked accuracy, failed to identify vehicle lanes, and offered limited or less practical classification categories. This study introduces a fine-tuned YOLOv11 model, trained on almost 800 bird's-eye view images, to enhance vehicle speed detection accuracy which is much higher compare to the previous works. The proposed system identifies the lane for each vehicle and classifies vehicles into two categories: cars and heavy vehicles. Designed to meet the specific requirements of traffic monitoring and regulation, the model also evaluates the effects of factors such as drone height, distance of Region of Interest (ROI), and vehicle speed on detection accuracy and speed measurement. Drone footage collected from Northern California was used to assess the proposed system. The fine-tuned YOLOv11 achieved its best performance with a mean absolute error (MAE) of 0.97 mph and mean squared error (MSE) of 0.94 $\text{mph}^2$, demonstrating its efficacy in addressing challenges in vehicle speed detection and classification.
加州车辆数量的增加,由于交通运输系统不足和测速摄像头分布稀疏,需要更有效的车辆速度检测方法。每条车道上的车辆速度监测对于监控高占有量车辆(HOV)车道的速度、区分具有不同限速的小型汽车与重型车辆以及执行重型车辆的车道限制至关重要。尽管先前的研究利用YOLO(You Only Look Once)进行车辆速度检测,但它们往往缺乏准确性,无法识别车辆所在的车道,并且提供的分类类别有限或实用性较差。 本研究引入了一种基于近800张鸟瞰图图像训练的微调版YOLOv11模型,以提高车辆速度检测精度,相较于以前的研究成果有显著提升。该系统可以为每辆汽车确定所在车道并将其归类为两类:小型汽车和重型车辆。为了满足交通监控与管理的具体需求,该模型还评估了无人机飞行高度、感兴趣区域(ROI)的距离以及车速等因素对检测准确性和速度测量的影响。 研究使用从加州北部收集的无人机视频来测试所提出的系统。微调后的YOLOv11在均方误差(MSE)为0.94 mph²和平均绝对误差(MAE)为0.97 mph时表现最佳,这表明该模型能够有效解决车辆速度检测及分类中的挑战。
https://arxiv.org/abs/2506.11239
This paper presents a lightweight and energy-efficient object detection solution for aerial imagery captured during emergency response situations. We focus on deploying the YOLOv4-Tiny model, a compact convolutional neural network, optimized through post-training quantization to INT8 precision. The model is trained on a custom-curated aerial emergency dataset, consisting of 10,820 annotated images covering critical emergency scenarios. Unlike prior works that rely on publicly available datasets, we created this dataset ourselves due to the lack of publicly available drone-view emergency imagery, making the dataset itself a key contribution of this work. The quantized model is evaluated against YOLOv5-small across multiple metrics, including mean Average Precision (mAP), F1 score, inference time, and model size. Experimental results demonstrate that the quantized YOLOv4-Tiny achieves comparable detection performance while reducing the model size from 22.5 MB to 6.4 MB and improving inference speed by 44\%. With a 71\% reduction in model size and a 44\% increase in inference speed, the quantized YOLOv4-Tiny model proves highly suitable for real-time emergency detection on low-power edge devices.
本文提出了一种轻量且节能的对象检测解决方案,专门用于紧急响应情况下捕获的航拍图像。我们专注于部署经过后训练量化到INT8精度优化后的YOLOv4-Tiny模型,这是一种紧凑型卷积神经网络。该模型是在一个定制的、针对空中应急场景的数据集上进行训练的,这个数据集中包含10,820张注释过的图片,涵盖了关键的紧急情况。与依赖公开可用数据集的先前工作不同,由于缺乏可获得的无人机视角下的紧急图像资源,我们自己创建了这一数据集,这使得该数据集本身成为了本文的重要贡献之一。 通过多个指标(包括平均精度均值(mAP)、F1分数、推理时间和模型大小)对量化后的模型与YOLOv5-small进行了评估。实验结果表明,量化后的YOLOv4-Tiny在检测性能上达到了与后者相当的水平,同时将模型大小从22.5MB减少到了6.4MB,并提升了44%的推理速度。凭借71%的模型体积缩减和44%的推理速度提升,量化后的YOLOv4-Tiny模型被证明非常适合在低功耗边缘设备上进行实时紧急情况检测。
https://arxiv.org/abs/2506.09299
MOMAV (Marco's Omnidirectional Micro Aerial Vehicle) is a multirotor drone that is fully actuated, meaning it can control its orientation independently of its position. MOMAV is also highly symmetrical, making its flight efficiency largely unaffected by its current orientation. These characteristics are achieved by a novel drone design where six rotor arms align with the vertices of an octahedron, and where each arm can actively rotate along its long axis. Various standout features of MOMAV are presented: The high flight efficiency compared to arm configuration of other fully-actuated drones, the design of an original rotating arm assembly featuring slip-rings used to enable continuous arm rotation, and a novel control allocation algorithm based on sequential quadratic programming (SQP) used to calculate throttle and arm-angle setpoints in flight. Flight tests have shown that MOMAV is able to achieve remarkably low mean position/orientation errors of 6.6mm, 2.1° ({\sigma}: 3.0mm, 1.0°) when sweeping position setpoints, and 11.8mm, 3.3° ({\sigma}: 8.6mm, 2.0°) when sweeping orientation setpoints.
MOMAV(Marco的全方位微型空中飞行器)是一款全动型多旋翼无人机,它能够独立控制其姿态而不受位置的影响。MOMAV还具有高度对称性,这意味着它的飞行效率不会因当前的姿态而显著改变。这一特点通过一种独特的设计实现:六个旋翼臂与正八面体的顶点相匹配,并且每个臂都能沿其长轴主动旋转。 MOMAV 的几个突出特性如下: 1. 相较于其他全动型无人机,它具有更高的飞行效率。 2. 设计了一个创新的旋转臂组件,该组件采用了滑环以实现持续的臂部旋转。 3. 基于序列二次规划(SQP)算法开发了一种新颖的姿态分配算法,在飞行中用于计算油门和臂角设定值。 飞行测试表明,当在位置设置点进行扫掠时,MOMAV能够达到平均位置/姿态误差6.6毫米、2.1度(标准差:3.0毫米、1.0度);而在姿态设置点进行扫掠时,其误差为11.8毫米、3.3度(标准差:8.6毫米、2.0度)。这些结果显示了MOMAV的卓越性能。
https://arxiv.org/abs/2506.08868
Reinforcement learning (RL) algorithms can find an optimal policy for a single agent to accomplish a particular task. However, many real-world problems require multiple agents to collaborate in order to achieve a common goal. For example, a robot executing a task in a warehouse may require the assistance of a drone to retrieve items from high shelves. In Decentralized Multi-Agent RL (DMARL), agents learn independently and then combine their policies at execution time, but often must satisfy constraints on compatibility of local policies to ensure that they can achieve the global task when combined. In this paper, we study how providing high-level symbolic knowledge to agents can help address unique challenges of this setting, such as privacy constraints, communication limitations, and performance concerns. In particular, we extend the formal tools used to check the compatibility of local policies with the team task, making decentralized training with theoretical guarantees usable in more scenarios. Furthermore, we empirically demonstrate that symbolic knowledge about the temporal evolution of events in the environment can significantly expedite the learning process in DMARL.
强化学习(RL)算法能够为单一智能体找到完成特定任务的最优策略。然而,许多现实世界的问题需要多个智能体协作以达成共同目标。例如,在仓库中执行任务的机器人可能需要无人机的帮助来从高架上取回物品。在分散式多代理强化学习(DMARL)中,各个智能体独立学习并在执行时结合各自的策略,但通常必须满足关于局部策略兼容性的约束条件,以确保它们可以实现整体目标。 在这篇论文中,我们研究了向各智能体提供高层次的符号知识如何能帮助解决这一设置中的独特挑战,例如隐私限制、通信限制和性能问题。具体而言,我们将用于检查局部策略与团队任务兼容性的形式化工具进行了扩展,使得带有理论保证的分散式训练在更多场景中得以应用。 此外,我们还通过实验证明了关于环境中事件时间演变的知识可以显著加快DMARL中的学习过程。
https://arxiv.org/abs/2506.07829
Current robotic agents, such as autonomous vehicles (AVs) and drones, need to deal with uncertain real-world environments with appropriate situational awareness (SA), risk awareness, coordination, and decision-making. The SymAware project strives to address this issue by designing an architecture for artificial awareness in multi-agent systems, enabling safe collaboration of autonomous vehicles and drones. However, these agents will also need to interact with human users (drivers, pedestrians, drone operators), which in turn requires an understanding of how to model the human in the interaction scenario, and how to foster trust and transparency between the agent and the human. In this work, we aim to create a data-driven model of a human driver to be integrated into our SA architecture, grounding our research in the principles of trustworthy human-agent interaction. To collect the data necessary for creating the model, we conducted a large-scale user-centered study on human-AV interaction, in which we investigate the interaction between the AV's transparency and the users' behavior. The contributions of this paper are twofold: First, we illustrate in detail our human-AV study and its findings, and second we present the resulting Markov chain models of the human driver computed from the study's data. Our results show that depending on the AV's transparency, the scenario's environment, and the users' demographics, we can obtain significant differences in the model's transitions.
当前的机器人代理,如自动驾驶汽车(AVs)和无人机,在面对不确定的真实世界环境时需要具备适当的情境感知(SA)、风险意识、协调与决策能力。SymAware项目旨在通过为多智能体系统设计一个人工意识架构来解决这一问题,从而实现自主车辆和无人机的安全协作。然而,这些代理还需要与人类用户(驾驶员、行人、无人机操作员)进行交互,这反过来又需要理解如何在互动场景中建模人类行为,并且如何促进代理人与人类之间的信任与透明度。在这项工作中,我们旨在创建一个基于数据驱动的人类驾驶员模型,将其集成到我们的SA架构中,以此为基础研究可信赖的人机交互原则。 为了收集构建该模型所需的数据,我们开展了一项大规模以用户为中心的研究,考察了自主车辆的透明性与其用户的互动行为之间的关系。本论文的主要贡献有两方面:首先,详细阐述了我们在人类-自动驾驶汽车(AV)互动中的研究及其发现;其次,根据研究所获得数据,呈现计算得出的人类驾驶员马尔可夫链模型。我们的研究成果表明,根据自动驾驶汽车的透明度、场景环境以及用户的个人背景特征的不同,我们能够观察到该模型转换的显著差异。
https://arxiv.org/abs/2506.07633
We present a real-time system for per-tree canopy volume estimation using mobile LiDAR data collected during routine robotic navigation. Unlike prior approaches that rely on static scans or assume uniform orchard structures, our method adapts to varying field geometries via an integrated pipeline of LiDAR-inertial odometry, adaptive segmentation, and geometric reconstruction. We evaluate the system across two commercial orchards, one pistachio orchard with regular spacing and one almond orchard with dense, overlapping crowns. A hybrid clustering strategy combining DBSCAN and spectral clustering enables robust per-tree segmentation, achieving 93% success in pistachio and 80% in almond, with strong agreement to drone derived canopy volume estimates. This work advances scalable, non-intrusive tree monitoring for structurally diverse orchard environments.
我们提出了一种实时系统,用于利用在机器人常规导航过程中采集的移动LiDAR数据来进行单棵树木冠层体积估算。与依赖静态扫描或假设果园结构均匀的传统方法不同,我们的方法通过集成的里程测量(LiDAR-惯性)、自适应分割和几何重建管道,能够适应变化多样的田野几何形状。我们在两个商业果园中评估了该系统的效果:一个是规则间距的开心果果园,另一个是密集、冠层重叠的杏仁果园。我们采用了一种结合DBSCAN与谱聚类的混合聚类策略,实现了鲁棒性的单树分割,在开心果果园中的成功率为93%,在杏仁果园中为80%;并且我们的估算结果与无人机获取的冠层体积高度一致。这项工作推动了结构多样化的果园环境中可扩展且非侵入式的树木监测技术的发展。
https://arxiv.org/abs/2506.08061
Recent advances in agentic and physical artificial intelligence (AI) have largely focused on ground-based platforms such as humanoid and wheeled robots, leaving aerial robots relatively underexplored. Meanwhile, state-of-the-art unmanned aerial vehicle (UAV) multimodal vision-language systems typically rely on closed-source models accessible only to well-resourced organizations. To democratize natural language control of autonomous drones, we present an open-source agentic framework that integrates PX4-based flight control, Robot Operating System 2 (ROS 2) middleware, and locally hosted models using Ollama. We evaluate performance both in simulation and on a custom quadcopter platform, benchmarking four large language model (LLM) families for command generation and three vision-language model (VLM) families for scene understanding.
近期,代理型和物理人工智能(AI)领域的进展主要集中在地面平台,如人形机器人和轮式机器人上,而空中机器人的研究相对较少。同时,最先进的无人驾驶飞行器(UAV)多模态视觉-语言系统通常依赖于仅对资源充足的组织开放的闭源模型。为了使自主无人机的自然语言控制普及化,我们推出了一种开源代理框架,该框架结合了基于PX4的飞行控制系统、Robot Operating System 2 (ROS 2) 中间件以及使用Ollama本地托管的模型。我们在模拟环境和定制四旋翼平台中评估了性能,并针对指令生成对四种大型语言模型(LLM)家族进行了基准测试,同时针对场景理解对三种视觉-语言模型(VLM)家族进行了基准测试。
https://arxiv.org/abs/2506.07509
Diffusion models have shown remarkable flexibility for solving inverse problems without task-specific retraining. However, existing approaches such as Manifold Preserving Guided Diffusion (MPGD) apply only a single gradient update per denoising step, limiting restoration fidelity and robustness, especially in embedded or out-of-distribution settings. In this work, we introduce a multistep optimization strategy within each denoising timestep, significantly enhancing image quality, perceptual accuracy, and generalization. Our experiments on super-resolution and Gaussian deblurring demonstrate that increasing the number of gradient updates per step improves LPIPS and PSNR with minimal latency overhead. Notably, we validate this approach on a Jetson Orin Nano using degraded ImageNet and a UAV dataset, showing that MPGD, originally trained on face datasets, generalizes effectively to natural and aerial scenes. Our findings highlight MPGD's potential as a lightweight, plug-and-play restoration module for real-time visual perception in embodied AI agents such as drones and mobile robots.
扩散模型在解决逆向问题时表现出惊人的灵活性,无需针对特定任务进行重新训练。然而,现有的方法如保流形引导扩散(MPGD)仅在每次去噪步骤中应用一次梯度更新,这限制了恢复的精度和鲁棒性,尤其是在嵌入式或分布外设置中更为明显。在这项工作中,我们介绍了一种在每个去噪时间步内进行多步优化策略的方法,从而显著提高了图像质量、感知准确性和泛化能力。我们在超分辨率和高斯模糊去除上的实验表明,增加每一步的梯度更新次数可以提高LPIPS(Learned Perceptual Image Patch Similarity)和PSNR(峰值信噪比),同时仅增加了微小的时间延迟开销。 值得注意的是,我们使用Jetson Orin Nano设备在降级后的ImageNet数据集和无人机数据集上验证了这一方法的有效性。实验表明,最初在面部数据集中训练的MPGD模型可以有效地推广到自然场景和空中图像中。我们的研究结果强调了MPGD作为轻量级、即插即用恢复模块用于实时视觉感知的潜力,特别是在如无人机和移动机器人等具身人工智能代理中的应用。
https://arxiv.org/abs/2506.07286
This work presents UNO, a unified monocular visual odometry framework that enables robust and adaptable pose estimation across diverse environments, platforms, and motion patterns. Unlike traditional methods that rely on deployment-specific tuning or predefined motion priors, our approach generalizes effectively across a wide range of real-world scenarios, including autonomous vehicles, aerial drones, mobile robots, and handheld devices. To this end, we introduce a Mixture-of-Experts strategy for local state estimation, with several specialized decoders that each handle a distinct class of ego-motion patterns. Moreover, we introduce a fully differentiable Gumbel-Softmax module that constructs a robust inter-frame correlation graph, selects the optimal expert decoder, and prunes erroneous estimates. These cues are then fed into a unified back-end that combines pre-trained, scale-independent depth priors with a lightweight bundling adjustment to enforce geometric consistency. We extensively evaluate our method on three major benchmark datasets: KITTI (outdoor/autonomous driving), EuRoC-MAV (indoor/aerial drones), and TUM-RGBD (indoor/handheld), demonstrating state-of-the-art performance.
这项工作提出了UNO,一个统一的单目视觉里程计框架,能够在不同的环境、平台和运动模式下实现稳健且适应性强的姿态估计。与传统的依赖于特定部署调整或预定义运动先验的方法不同,我们的方法能够广泛适用于各种现实场景,包括自主车辆、空中无人机、移动机器人以及手持设备。为此,我们引入了专家混合策略来进行局部状态估计,并设计了几种专门处理不同类型自运动模式的解码器。此外,我们还提出了一种完全可微化的Gumbel-Softmax模块来构建稳健的帧间相关性图,选择最优专家解码器并修剪错误估算结果。这些线索随后被输入到一个统一的后端中,该后端结合了预训练且尺度无关的深度先验和轻量级捆绑调整以强制执行几何一致性。我们在三个主要基准数据集上对我们的方法进行了广泛评估:KITTI(户外/自动驾驶)、EuRoC-MAV(室内/空中无人机)和TUM-RGBD(室内/手持设备),展示了最先进的性能水平。
https://arxiv.org/abs/2506.07013
Cooperative autonomous robotic systems have significant potential for executing complex multi-task missions across space, air, ground, and maritime domains. But they commonly operate in remote, dynamic and hazardous environments, requiring rapid in-mission adaptation without reliance on fragile or slow communication links to centralised compute. Fast, on-board replanning algorithms are therefore needed to enhance resilience. Reinforcement Learning shows strong promise for efficiently solving mission planning tasks when formulated as Travelling Salesperson Problems (TSPs), but existing methods: 1) are unsuitable for replanning, where agents do not start at a single location; 2) do not allow cooperation between agents; 3) are unable to model tasks with variable durations; or 4) lack practical considerations for on-board deployment. Here we define the Cooperative Mission Replanning Problem as a novel variant of multiple TSP with adaptations to overcome these issues, and develop a new encoder/decoder-based model using Graph Attention Networks and Attention Models to solve it effectively and efficiently. Using a simple example of cooperative drones, we show our replanner consistently (90% of the time) maintains performance within 10% of the state-of-the-art LKH3 heuristic solver, whilst running 85-370 times faster on a Raspberry Pi. This work paves the way for increased resilience in autonomous multi-agent systems.
合作自主机器人系统在执行跨空间、空中、地面和海洋领域的复杂多任务时具有巨大潜力。然而,这些系统通常在偏远、动态且危险的环境中运行,需要快速适应任务需求,而无需依赖脆弱或缓慢的通信链路与集中式计算资源。因此,需要高效的车载重规划算法来增强系统的鲁棒性。强化学习(Reinforcement Learning)显示出可以有效地解决作为旅行商问题(TSPs)形式化的问题,但在现有的方法中:1) 不适用于重新规划场景,在这种场景下代理不会从单一位置开始;2) 不能促进代理之间的合作;3) 无法模拟任务的可变持续时间;4) 缺乏适合车载部署的实际考虑。在这里,我们定义了“协作任务重规划问题”作为多个TSP的一个新变体,并进行了适应性改进以克服上述挑战。此外,我们开发了一种基于编码器/解码器模型的新方法,该模型结合图注意力网络(Graph Attention Networks)和注意力机制来有效且高效地解决这一问题。 通过一个简单的合作无人机示例,我们展示了我们的重规划算法能够在90%的时间内保持性能,在性能上仅比最先进的LKH3启发式求解器低10%,同时在树莓派设备上的运行速度提高了85到370倍。这项工作为提高自主多代理系统的鲁棒性铺平了道路。
https://arxiv.org/abs/2506.06094
The rapid detection of abnormal body temperatures in urban populations is essential for managing public health risks, especially during outbreaks of infectious diseases. Multi-drone thermal screening systems offer promising solutions for fast, large-scale, and non-intrusive human temperature monitoring. However, trajectory planning for multiple drones in complex urban environments poses significant challenges, including collision avoidance, coverage efficiency, and constrained flight environments. In this study, we propose an enhanced trust region sequential convex optimization (TR-SCO) algorithm for optimal trajectory planning of multiple drones performing thermal screening tasks. Our improved algorithm integrates a refined convex optimization formulation within a trust region framework, effectively balancing trajectory smoothness, obstacle avoidance, altitude constraints, and maximum screening coverage. Simulation results demonstrate that our approach significantly improves trajectory optimality and computational efficiency compared to conventional convex optimization methods. This research provides critical insights and practical contributions toward deploying efficient multi-drone systems for real-time thermal screening in urban areas. For reader who are interested in our research, we release our source code at this https URL.
在城市人群中快速检测异常体温对于管理公共卫生风险至关重要,特别是在传染病爆发期间。多无人机热成像筛查系统为实现快速、大规模且非侵入式的人体温度监测提供了有前景的解决方案。然而,在复杂的城市场景中规划多个无人机的飞行轨迹面临着重大挑战,包括碰撞避免、覆盖效率以及受限飞行环境等。在本研究中,我们提出了一种改进的信任区域序列凸优化(TR-SCO)算法,用于多架无人机进行热成像筛查任务时的最佳轨迹规划。我们的改进算法在一个信任区域内集成了精细的凸优化公式,有效地平衡了轨迹平滑度、障碍物避免、高度约束以及最大覆盖范围等因素。 仿真结果表明,与传统的凸优化方法相比,我们提出的方法在轨迹优化性和计算效率方面有显著提升。这项研究为部署高效的多无人机系统以实现在城市区域进行实时热成像筛查提供了关键见解和实用贡献。对于对我们的研究成果感兴趣的读者,我们在[这里](https://example.com)发布了源代码。 (请注意,URL部分的实际链接需要您自行提供或验证发布源代码的真实网址。)
https://arxiv.org/abs/2506.06012
Small, fast, and lightweight drones present significant challenges for traditional RGB cameras due to their limitations in capturing fast-moving objects, especially under challenging lighting conditions. Event cameras offer an ideal solution, providing high temporal definition and dynamic range, yet existing benchmarks often lack fine temporal resolution or drone-specific motion patterns, hindering progress in these areas. This paper introduces the Florence RGB-Event Drone dataset (FRED), a novel multimodal dataset specifically designed for drone detection, tracking, and trajectory forecasting, combining RGB video and event streams. FRED features more than 7 hours of densely annotated drone trajectories, using 5 different drone models and including challenging scenarios such as rain and adverse lighting conditions. We provide detailed evaluation protocols and standard metrics for each task, facilitating reproducible benchmarking. The authors hope FRED will advance research in high-speed drone perception and multimodal spatiotemporal understanding.
小型、快速且轻量级的无人机给传统的RGB相机带来了显著挑战,因为它们在捕捉高速移动对象方面存在局限性,尤其是在光照条件复杂的情况下。事件相机提供了一个理想的解决方案,能够提供高时间分辨率和动态范围,但现有的基准测试通常缺乏精细的时间分辨率或针对无人机特定运动模式的数据,这阻碍了相关领域的进步。本文介绍了佛罗伦萨RGB-事件无人机数据集(FRED),这是一个专门用于无人机检测、跟踪和轨迹预测的新型多模态数据集,结合了RGB视频流和事件流。FRED包含了超过7小时密集注释的无人机轨迹数据,使用了5种不同的无人机型号,并涵盖了具有挑战性的场景,例如雨天及不良光照条件。我们提供了每项任务详细的评估协议和标准指标,以促进可重复基准测试的进行。作者希望FRED能够推动高速无人机感知以及多模态时空理解的研究进展。
https://arxiv.org/abs/2506.05163
Despite significant advancements in dynamic neural rendering, existing methods fail to address the unique challenges posed by UAV-captured scenarios, particularly those involving monocular camera setups, top-down perspective, and multiple small, moving humans, which are not adequately represented in existing datasets. In this work, we introduce UAV4D, a framework for enabling photorealistic rendering for dynamic real-world scenes captured by UAVs. Specifically, we address the challenge of reconstructing dynamic scenes with multiple moving pedestrians from monocular video data without the need for additional sensors. We use a combination of a 3D foundation model and a human mesh reconstruction model to reconstruct both the scene background and humans. We propose a novel approach to resolve the scene scale ambiguity and place both humans and the scene in world coordinates by identifying human-scene contact points. Additionally, we exploit the SMPL model and background mesh to initialize Gaussian splats, enabling holistic scene rendering. We evaluated our method on three complex UAV-captured datasets: VisDrone, Manipal-UAV, and Okutama-Action, each with distinct characteristics and 10~50 humans. Our results demonstrate the benefits of our approach over existing methods in novel view synthesis, achieving a 1.5 dB PSNR improvement and superior visual sharpness.
尽管动态神经渲染技术取得了显著进展,但现有方法未能解决无人机(UAV)捕捉场景中的独特挑战,特别是单目相机设置、顶部视图视角以及多个小型移动人物的问题,这些问题在现有的数据集中并未得到充分表示。在此工作中,我们引入了UAV4D框架,该框架旨在实现由无人机捕获的真实世界动态场景的逼真渲染。 具体而言,我们解决了从单目视频数据中重建包含多个人行者的动态场景的挑战,并且无需额外传感器的支持就能完成这一任务。为此,我们将3D基础模型与人体网格重建模型相结合,用于同时重建场景背景和人物。我们提出了一种新的方法来解决场景尺度模糊问题,并通过识别人物-场景接触点将两者置于世界坐标系中。此外,利用SMPL(Skeletal Modeling and Parameterization)模型以及背景网格初始化高斯点集,这使整个场景的渲染成为可能。 我们在三个复杂的无人机捕捉数据集上评估了我们的方法:VisDrone、Manipal-UAV和Okutama-Action,每个数据集中包含10至50个个体,并且各具特色。实验结果表明,与现有方法相比,我们的方法在新颖视图合成方面表现出显著优势,实现了高达1.5 dB的PSNR(峰值信噪比)改进以及更佳的视觉清晰度。
https://arxiv.org/abs/2506.05011