Robots and other intelligent systems navigating in complex dynamic environments should predict future actions and intentions of surrounding agents to reach their goals efficiently and avoid collisions. The dynamics of those agents strongly depends on their tasks, roles, or observable labels. Class-conditioned motion prediction is thus an appealing way to reduce forecast uncertainty and get more accurate predictions for heterogeneous agents. However, this is hardly explored in the prior art, especially for mobile robots and in limited data applications. In this paper, we analyse different class-conditioned trajectory prediction methods on two datasets. We propose a set of conditional pattern-based and efficient deep learning-based baselines, and evaluate their performance on robotics and outdoors datasets (THÃR-MAGNI and Stanford Drone Dataset). Our experiments show that all methods improve accuracy in most of the settings when considering class labels. More importantly, we observe that there are significant differences when learning from imbalanced datasets, or in new environments where sufficient data is not available. In particular, we find that deep learning methods perform better on balanced datasets, but in applications with limited data, e.g., cold start of a robot in a new environment, or imbalanced classes, pattern-based methods may be preferable.
机器人和其他智能系统在复杂的动态环境中导航时,应预测周围代理的未来行动和意图,以高效地实现目标并避免碰撞。这些代理的动力学特性强烈依赖于它们的任务、角色或可观察标签。因此,基于类别的运动预测是一种减少预测不确定性并为异构代理获得更准确预测的有吸引力的方法。然而,在之前的研究中,这种方法对移动机器人以及在数据有限的应用领域中的探索还不够。 本文分析了两个数据集上不同的基于类别条件的轨迹预测方法,并提出了一组基于条件模式和高效的深度学习基线模型,并评估这些模型在机器人技术和户外数据集(THÖR-MAGNI 和 Stanford Drone 数据集)上的性能。我们的实验表明,在大多数情况下,考虑类标签时所有方法都能提高准确性。更重要的是,我们观察到从不平衡的数据集中学习或在没有足够数据的新环境中学习会有显著的不同。特别是,深度学习方法在平衡的数据集上表现更好,但在数据有限的应用场景中(例如,机器人进入新环境的初始阶段),或者面对类别不平衡的情况时,基于模式的方法可能更可取。
https://arxiv.org/abs/2510.03776
Rapid naloxone delivery via drones offers a promising solution for responding to opioid overdose emergencies (OOEs), by extending lifesaving interventions to medically untrained bystanders before emergency medical services (EMS) arrive. Recognizing the critical role of bystander situational awareness (SA) in human-autonomy teaming (HAT), we address a key research gap in real-time SA assessment by introducing the Drone-Assisted Naloxone Delivery Simulation Dataset (DANDSD). This pioneering dataset captures HAT during simulated OOEs, where college students without medical training act as bystanders tasked with administering intranasal naloxone to a mock overdose victim. Leveraging this dataset, we propose a video-based real-time SA assessment framework that utilizes graph embeddings and transformer models to assess bystander SA in real time. Our approach integrates visual perception and comprehension cues--such as geometric, kinematic, and interaction graph features--and achieves high-performance SA prediction. It also demonstrates strong temporal segmentation accuracy, outperforming the FINCH baseline by 9% in Mean over Frames (MoF) and 5% in Intersection over Union (IoU). This work supports the development of adaptive drone systems capable of guiding bystanders effectively, ultimately improving emergency response outcomes and saving lives.
通过无人机快速递送纳洛酮为应对阿片类药物过量紧急情况(OOEs)提供了一个有前景的解决方案,能够在紧急医疗服务到达之前将挽救生命的干预措施扩展到未接受医学培训的旁观者。鉴于情境意识(SA)在人-自主系统协作中的关键作用,我们通过引入无人机辅助纳洛酮递送模拟数据集(DANDSD),填补了实时SA评估这一重要研究空白。该开创性数据集记录了大学生志愿者在模拟OOEs中作为未接受医学训练的旁观者进行操作的情况,他们需要向假定的药物过量受害者施用鼻腔内纳洛酮。 利用这个数据集,我们提出了一种基于视频的实时SA评估框架,采用图嵌入和转换模型来评估旁观者的实时情境意识。我们的方法整合了视觉感知和理解线索——包括几何、运动和交互图形特征,并实现了高精度的情境意识预测。此外,它还展示了强大的时间分割准确度,在均帧(MoF)指标上比FINCH基线高出9%,在交并比(IoU)指标上高出5%。 这项工作支持开发能够有效引导旁观者适应的无人机系统,最终改善紧急响应结果,并挽救生命。
https://arxiv.org/abs/2510.03558
We propose SketchPlan, a diffusion-based planner that interprets 2D hand-drawn sketches over depth images to generate 3D flight paths for drone navigation. SketchPlan comprises two components: a SketchAdapter that learns to map the human sketches to projected 2D paths, and DiffPath, a diffusion model that infers 3D trajectories from 2D projections and a first person view depth image. Our model achieves zero-shot sim-to-real transfer, generating accurate and safe flight paths in previously unseen real-world environments. To train the model, we build a synthetic dataset of 32k flight paths using a diverse set of photorealistic 3D Gaussian Splatting scenes. We automatically label the data by computing 2D projections of the 3D flight paths onto the camera plane, and use this to train the DiffPath diffusion model. However, since real human 2D sketches differ significantly from ideal 2D projections, we additionally label 872 of the 3D flight paths with real human sketches and use this to train the SketchAdapter to infer the 2D projection from the human sketch. We demonstrate SketchPlan's effectiveness in both simulated and real-world experiments, and show through ablations that training on a mix of human labeled and auto-labeled data together with a modular design significantly boosts its capabilities to correctly interpret human intent and infer 3D paths. In real-world drone tests, SketchPlan achieved 100\% success in low/medium clutter and 40\% in unseen high-clutter environments, outperforming key ablations by 20-60\% in task completion.
我们提出了SketchPlan,这是一种基于扩散模型的规划器,它能够解析深度图像上的二维手绘草图以生成无人机导航的三维飞行路径。SketchPlan 包含两个组成部分:一个名为 SketchAdapter 的组件,用于学习将人类绘制的草图映射到投影的二维路径;另一个是 DiffPath,这是一个从二维投影和第一人称视角深度图像推断三维轨迹的扩散模型。 我们的模型实现了零样本仿真到现实的迁移能力,在以前未见过的真实世界环境中生成准确且安全的飞行路径。为了训练该模型,我们构建了一个包含32,000条合成飞行路径的数据集,这些数据是基于多样化的逼真三维高斯点云场景创建的。通过计算三维飞行路径在相机平面上的二维投影来自动标注数据,并利用这些标注来训练 DiffPath 扩散模型。 然而,由于实际的人类绘制的二维草图与理想的二维投影存在显著差异,我们还对其中872条三维飞行路径使用真实人类草图进行人工标注,并用此来训练 SketchAdapter 以从人类草图中推断出二维投影。我们在模拟和现实世界实验中展示了 SketchPlan 的有效性,并通过消融研究证明,在混合了人类标签数据和自动标签数据的培训下,以及采用模块化设计的方式能够显著提升其准确解读人类意图并推断三维路径的能力。 在真实世界的无人机测试中,SketchPlan 在低/中等复杂度环境中实现了100%的成功率,并且在未见过的高复杂度环境下达到了40%的成功率。与关键消融版本相比,在任务完成方面,SketchPlan 的表现要高出20%-60%。
https://arxiv.org/abs/2510.03545
Tiny flying robots hold great potential for search-and-rescue, safety inspections, and environmental monitoring, but their small size limits conventional sensing-especially with poor-lighting, smoke, dust or reflective obstacles. Inspired by nature, we propose a lightweight, 3.2-gram, whisker-based tactile sensing apparatus for tiny drones, enabling them to navigate and explore through gentle physical interaction. Just as rats and moles use whiskers to perceive surroundings, our system equips drones with tactile perception in flight, allowing obstacle sensing even in pitch-dark conditions. The apparatus uses barometer-based whisker sensors to detect obstacle locations while minimising destabilisation. To address sensor noise and drift, we develop a tactile depth estimation method achieving sub-6 mm accuracy. This enables drones to navigate, contour obstacles, and explore confined spaces solely through touch-even in total darkness along both soft and rigid surfaces. Running fully onboard a 192-KB RAM microcontroller, the system supports autonomous tactile flight and is validated in both simulation and real-world tests. Our bio-inspired approach redefines vision-free navigation, opening new possibilities for micro aerial vehicles in extreme environments.
小型飞行机器人在搜索救援、安全检查和环境监测方面具有巨大潜力,但其小尺寸限制了传统传感器的使用,特别是在光线昏暗、烟雾、尘埃或反射性障碍物的情况下。受自然启发,我们提出了一种轻量级、仅重3.2克、基于触须的触觉传感装置,专为微型无人机设计,使它们能够通过温和的身体互动导航和探索。就像老鼠和鼹鼠使用胡须来感知周围环境一样,我们的系统让无人机在飞行中具备了触觉感知能力,在完全黑暗的情况下也能进行障碍物检测。 该装置采用基于气压的触须传感器来探测障碍物的位置,同时尽量减少对无人机稳定性的干扰。为了解决传感器噪声和漂移问题,我们开发了一种触觉深度估计方法,能够实现亚6毫米的精度。这使得无人机能够在完全黑暗中通过触摸导航、绕过障碍物,并探索狭窄的空间——无论是柔软还是坚硬表面。 整个系统在仅配备192KB RAM的微控制器上运行,支持自主触控飞行,并已在模拟和现实世界测试中得到了验证。我们的仿生方法重新定义了无视觉导航的可能性,为微型飞行器在极端环境中的应用开辟了新的可能性。
https://arxiv.org/abs/2510.03119
Cross-modal drone navigation remains a challenging task in robotics, requiring efficient retrieval of relevant images from large-scale databases based on natural language descriptions. The RoboSense 2025 Track 4 challenge addresses this challenge, focusing on robust, natural language-guided cross-view image retrieval across multiple platforms (drones, satellites, and ground cameras). Current baseline methods, while effective for initial retrieval, often struggle to achieve fine-grained semantic matching between text queries and visual content, especially in complex aerial scenes. To address this challenge, we propose a two-stage retrieval refinement method: Caption-Guided Retrieval System (CGRS) that enhances the baseline coarse ranking through intelligent reranking. Our method first leverages a baseline model to obtain an initial coarse ranking of the top 20 most relevant images for each query. We then use Vision-Language-Model (VLM) to generate detailed captions for these candidate images, capturing rich semantic descriptions of their visual content. These generated captions are then used in a multimodal similarity computation framework to perform fine-grained reranking of the original text query, effectively building a semantic bridge between the visual content and natural language descriptions. Our approach significantly improves upon the baseline, achieving a consistent 5\% improvement across all key metrics (Recall@1, Recall@5, and Recall@10). Our approach win TOP-2 in the challenge, demonstrating the practical value of our semantic refinement strategy in real-world robotic navigation scenarios.
跨模态无人机导航在机器人领域仍是一项具有挑战性的任务,需要根据自然语言描述从大规模数据库中高效检索相关图像。RoboSense 2025 Track 4 挑战赛致力于解决这一问题,重点在于多平台(无人机、卫星和地面相机)上基于自然语言指导的跨视角图像检索。尽管现有的基准方法在初步检索方面效果良好,但它们往往难以实现文本查询与视觉内容之间细微语义匹配,尤其是在复杂的空中场景中。 为了解决这一挑战,我们提出了一种两阶段检索优化方法:Caption-Guided Retrieval System(CGRS)。该系统通过智能重排名改进了基准粗排的结果。具体来说,我们的方法首先利用一个基准模型获取每个查询的前20张最相关图像的初步粗略排序。接着,使用视觉-语言模型(VLM)为这些候选图像生成详细的描述性字幕,捕捉它们视觉内容中的丰富语义信息。随后,在多模态相似度计算框架下利用这些生成的字幕对原始文本查询进行精细重排名,有效地在视觉内容和自然语言描述之间建立了一座语义桥梁。 我们的方法显著超越了基准模型,在所有关键指标(Recall@1、Recall@5 和 Recall@10)上均实现了平均5%的性能提升。我们在挑战赛中获得了第二名的成绩,这充分展示了我们提出的语义优化策略在现实世界中的机器人导航场景中的实用价值。 通过上述方法和实验结果,可以发现基于生成性描述与多模态相似度计算的重排名机制能够有效增强图像检索模型的语义理解能力,在实际应用中具有重要的参考意义。
https://arxiv.org/abs/2510.02728
The growing integration of drones across commercial, industrial, and civilian domains has introduced significant cybersecurity challenges, particularly due to the susceptibility of drone networks to a wide range of cyberattacks. Existing intrusion detection mechanisms often lack the adaptability, efficiency, and generalizability required for the dynamic and resource constrained environments in which drones operate. This paper proposes TSLT-Net, a novel lightweight and unified Temporal Spatial Transformer based intrusion detection system tailored specifically for drone networks. By leveraging self attention mechanisms, TSLT-Net effectively models both temporal patterns and spatial dependencies in network traffic, enabling accurate detection of diverse intrusion types. The framework includes a streamlined preprocessing pipeline and supports both multiclass attack classification and binary anomaly detection within a single architecture. Extensive experiments conducted on the ISOT Drone Anomaly Detection Dataset, consisting of more than 2.3 million labeled records, demonstrate the superior performance of TSLT-Net with 99.99 percent accuracy in multiclass detection and 100 percent in binary anomaly detection, while maintaining a minimal memory footprint of only 0.04 MB and 9722 trainable parameters. These results establish TSLT-Net as an effective and scalable solution for real time drone cybersecurity, particularly suitable for deployment on edge devices in mission critical UAV systems.
无人机在商业、工业和民用领域的日益普及,带来了显著的网络安全挑战,特别是在无人机网络容易遭受各种类型的网络攻击的情况下。现有的入侵检测机制往往缺乏适应性、效率以及适用于动态且资源受限环境所需的泛化能力。本文提出了一种新型轻量级统一的时间空间变换器(Temporal Spatial Transformer)基于的入侵检测系统TSLT-Net,该系统专门针对无人机网络设计。 通过利用自注意力机制,TSLT-Net能够有效地建模网络流量中的时间模式和空间依赖性,从而实现对多种入侵类型的准确检测。此框架包含一个精简的数据预处理流水线,并且在同一架构中支持多类攻击分类和二元异常检测。在ISOT无人机异常检测数据集上进行的大量实验(该数据集包括超过230万条标注记录)表明,TSLT-Net在多类检测中表现出99.99%的准确率,在二元异常检测中的准确率为100%,同时保持了仅40 KB内存占用量和9722个可训练参数的小型化特征。这些结果证明了TSLT-Net作为实时无人机网络安全的有效且可扩展解决方案,特别适合部署在任务关键性的UAV系统边缘设备上。
https://arxiv.org/abs/2510.02711
This paper introduces a novel benchmark dataset of Visible and Near-Infrared (VNIR) hyperspectral imagery acquired via an unmanned aerial vehicle (UAV) platform for landmine and unexploded ordnance (UXO) detection research. The dataset was collected over a controlled test field seeded with 143 realistic surrogate landmine and UXO targets, including surface, partially buried, and fully buried configurations. Data acquisition was performed using a Headwall Nano-Hyperspec sensor mounted on a multi-sensor drone platform, flown at an altitude of approximately 20.6 m, capturing 270 contiguous spectral bands spanning 398-1002 nm. Radiometric calibration, orthorectification, and mosaicking were performed followed by reflectance retrieval using a two-point Empirical Line Method (ELM), with reference spectra acquired using an SVC spectroradiometer. Cross-validation against six reference objects yielded RMSE values below 1.0 and SAM values between 1 and 6 degrees in the 400-900 nm range, demonstrating high spectral fidelity. The dataset is released alongside raw radiance cubes, GCP/AeroPoint data, and reference spectra to support reproducible research. This contribution fills a critical gap in open-access UAV-based hyperspectral data for landmine detection and offers a multi-sensor benchmark when combined with previously published drone-based electromagnetic induction (EMI) data from the same test field.
这篇论文介绍了一个新颖的基准数据集,该数据集包含了通过无人机(UAV)平台获取的可见光和近红外(VNIR)高光谱图像,用于地雷和未爆炸弹药(UXO)检测研究。数据集是在一个受控测试场上收集的,该场地播种了143个现实的地雷和UXO替代目标,包括表面、部分埋藏和完全埋藏配置。 数据采集使用的是安装在多传感器无人机平台上的Headwall Nano-Hyperspec传感器,在大约20.6米的高度飞行,并捕捉到从398-1002纳米的270个连续光谱波段。随后进行了辐射校准、正射纠正和镶嵌处理,然后利用两点多经验线法(ELM)检索反射率,参考光谱则使用SVC分光辐射计获取。与六个参考对象进行交叉验证后,在400-900纳米范围内的RMSE值低于1.0,SAM值在1到6度之间,证明了该数据集具有很高的光谱准确性。 除了原始辐射立方体、GCP/AeroPoint数据和参考光谱之外,本研究还发布了上述数据集,以支持可重复的研究。此贡献填补了开放获取的基于无人机的高光谱数据用于地雷探测的关键空白,并在与之前发布的同一测试场地的基于无人机的电磁感应(EMI)数据结合使用时,提供了多传感器基准。
https://arxiv.org/abs/2510.02700
Density map estimation can be used to estimate object counts in dense and occluded scenes where discrete counting-by-detection methods fail. We propose a multicategory counting framework that leverages a Twins pyramid vision-transformer backbone and a specialised multi-class counting head built on a state-of-the-art multiscale decoding approach. A two-task design adds a segmentation-based Category Focus Module, suppressing inter-category cross-talk at training time. Training and evaluation on the VisDrone and iSAID benchmarks demonstrates superior performance versus prior multicategory crowd-counting approaches (33%, 43% and 64% reduction to MAE), and the comparison with YOLOv11 underscores the necessity of crowd counting methods in dense scenes. The method's regional loss opens up multi-class crowd counting to new domains, demonstrated through the application to a biodiversity monitoring dataset, highlighting its capacity to inform conservation efforts and enable scalable ecological insights.
密度图估计可以用来估算在密集且遮挡场景中的物体数量,这种情况下离散的检测计数方法会失效。我们提出了一种多类别计数框架,该框架利用了Twins金字塔视觉变换器骨干网络以及一种基于最先进的多尺度解码方法构建的专业化多类计数头部。通过添加一个基于分割的Category Focus模块(任务二),设计成两阶段的任务架构可以在训练期间抑制跨类别间的相互干扰。 在VisDrone和iSAID基准数据集上的训练和评估显示,相比之前的多类别人群计数方法,该框架表现出更优越的性能(平均绝对误差(MAE)分别降低了33%,43%和64%)。与YOLOv11的比较进一步强调了在密集场景中采用人群计数方法的重要性。 此外,该方法引入的区域损失函数为多类别人群计数打开了新的应用领域。这一优势通过将其应用于一个生物多样性监测数据集得以展示,突显了其在指导保护工作和促进可扩展生态学洞察方面的能力。
https://arxiv.org/abs/2510.02213
This paper addresses a critical aerial defense challenge in contested airspace, involving three autonomous aerial vehicles -- a hostile drone (the pursuer), a high-value drone (the evader), and a protective drone (the defender). We present a cooperative guidance framework for the evader-defender team that guarantees interception of the pursuer before it can capture the evader, even under highly dynamic and uncertain engagement conditions. Unlike traditional heuristic, optimal control, or differential game-based methods, we approach the problem within a time-constrained guidance framework, leveraging true proportional navigation based approach that ensures robust and guaranteed solutions to the aerial defense problem. The proposed strategy is computationally lightweight, scalable to a large number of agent configurations, and does not require knowledge of the pursuer's strategy or control laws. From arbitrary initial geometries, our method guarantees that key engagement errors are driven to zero within a fixed time, leading to a successful mission. Extensive simulations across diverse and adversarial scenarios confirm the effectiveness of the proposed strategy and its relevance for real-time autonomous defense in contested airspace environments.
本文探讨了在争议空域中的一项关键空中防御挑战,涉及三个自主飞行器:一个敌对无人机(追击者),一个高价值无人机(躲避者)和一个保护性无人机(保卫者)。我们提出了一种合作引导框架,旨在确保在动态且不确定的交战条件下,由躲避者和保卫者组成的团队能够拦截追击者,从而防止其捕获躲避者。与传统的基于启发式、最优控制或微分博弈的方法不同,我们的方法在一个受时间约束的引导框架内解决问题,并利用真正的比例导航方法来确保空中防御问题具有鲁棒性和确定性的解决方案。提出的策略计算成本较低,可扩展到大量代理配置中,并且不需要了解追击者的策略或控制规律。 从任意初始几何位置开始,该方法保证在固定时间内将关键交战误差驱动至零,从而实现成功的任务目标。通过多样化和对抗性场景的广泛模拟验证了所提出策略的有效性和其对于实时自主防御在争议空域环境中的相关性。
https://arxiv.org/abs/2510.02087
A comprehensive understanding of animal behavior ecology depends on scalable approaches to quantify and interpret complex, multidimensional behavioral patterns. Traditional field observations are often limited in scope, time-consuming, and labor-intensive, hindering the assessment of behavioral responses across landscapes. To address this, we present kabr-tools (Kenyan Animal Behavior Recognition Tools), an open-source package for automated multi-species behavioral monitoring. This framework integrates drone-based video with machine learning systems to extract behavioral, social, and spatial metrics from wildlife footage. Our pipeline leverages object detection, tracking, and behavioral classification systems to generate key metrics, including time budgets, behavioral transitions, social interactions, habitat associations, and group composition dynamics. Compared to ground-based methods, drone-based observations significantly improved behavioral granularity, reducing visibility loss by 15% and capturing more transitions with higher accuracy and continuity. We validate kabr-tools through three case studies, analyzing 969 behavioral sequences, surpassing the capacity of traditional methods for data capture and annotation. We found that, like Plains zebras, vigilance in Grevy's zebras decreases with herd size, but, unlike Plains zebras, habitat has a negligible impact. Plains and Grevy's zebras exhibit strong behavioral inertia, with rare transitions to alert behaviors and observed spatial segregation between Grevy's zebras, Plains zebras, and giraffes in mixed-species herds. By enabling automated behavioral monitoring at scale, kabr-tools offers a powerful tool for ecosystem-wide studies, advancing conservation, biodiversity research, and ecological monitoring.
动物行为生态学的全面理解依赖于可扩展的方法来量化和解释复杂的多维度行为模式。传统的野外观察方法通常受限于范围、耗时且劳动密集,这阻碍了跨景观的行为反应评估。为了解决这一问题,我们提出了一套开源工具包kabr-tools(肯尼亚动物行为识别工具),用于自动化多种物种的行为监测。该框架结合了基于无人机的视频与机器学习系统,从野生动物录像中提取行为、社会和空间指标。我们的工作流程利用目标检测、跟踪及行为分类系统生成关键指标,包括时间预算、行为转换、社会互动、栖息地关联以及群体组成动态。 相较于地面方法,基于无人机的观察显著提升了行为细节度,减少了15%的能见度损失,并以更高的准确性和连续性捕捉到更多的转变。我们通过三个案例研究验证了kabr-tools的有效性,分析了969个行为序列,超过了传统方法在数据捕获和注释方面的能力。 我们的研究发现,与大草原斑马一样,格雷氏斑马的警戒行为随群体大小增加而减少;但与大草原斑马不同的是,栖息地对其影响可以忽略不计。两种斑马都表现出强烈的行为惯性,很少有向警觉状态转变的情况,并且在多物种群落中观察到格雷氏斑马、大草原斑马和长颈鹿之间的空间隔离。 通过支持大规模自动化行为监测,kabr-tools为生态系统范围内的研究提供了一种强大的工具,推动了保护工作、生物多样性研究及生态监控的发展。
https://arxiv.org/abs/2510.02030
When a single pilot is responsible for managing a multi-drone system, the task demands varying levels of autonomy, from direct control of individual UAVs, to group-level coordination, to fully autonomous swarm behaviors for accomplishing high-level tasks. Enabling such flexible interaction requires a framework that supports multiple modes of shared autonomy. As language models continue to improve in reasoning and planning, they provide a natural foundation for such systems, reducing pilot workload by enabling high-level task delegation through intuitive, language-based interfaces. In this paper we present TACOS (Task-Agnostic COordinator of a multi-drone System), a unified framework that enables high-level natural language control of multi-UAV systems through Large Language Models (LLMs). TACOS integrates three key capabilities into a single architecture: a one-to-many natural language interface for intuitive user interaction, an intelligent coordinator for translating user intent into structured task plans, and an autonomous agent that executes plans interacting with the real-world. TACOS allows a LLM to interact with a library of executable APIs, bridging semantic reasoning with real-time multi-robot coordination. We demonstrate the system in real-world multi-drone system and conduct an ablation study to assess the contribution of each module.
当一名飞行员负责管理一个多无人机系统时,任务需要不同级别的自主权,从单独控制每个无人飞行器(UAV),到群体层面的协调,再到为完成高层次任务而采用完全自主的群集行为。为了实现这种灵活互动,需要一个支持多种共享自主模式的框架。随着语言模型在推理和规划方面持续改进,它们提供了此类系统的一个自然基础,通过直观的语言界面实现高级别任务委派,从而减轻飞行员的工作负担。 本文介绍了TACOS(多无人机系统的无任务特异性协调器),这是一个统一框架,它通过大型语言模型(LLMs)使多UAV系统的高层次自然语言控制成为可能。TACOS将三种关键能力整合到一个架构中:一种直观的用户交互的一对多自然语言接口、一位智能协调员用于翻译用户的意图为结构化的任务计划以及一个自主代理执行与现实世界互动的任务规划。TACOS使LLM能够与可执行API库进行交互,连接语义推理和实时多机器人协同作业。 我们通过实际的多无人机系统展示了该系统的应用,并进行了删减研究来评估每个模块对整体性能的贡献。
https://arxiv.org/abs/2510.01869
Humanoid robot soccer presents several challenges, particularly in maintaining system stability during aggressive kicking motions while achieving precise ball trajectory control. Current solutions, whether traditional position-based control methods or reinforcement learning (RL) approaches, exhibit significant limitations. Model predictive control (MPC) is a prevalent approach for ordinary quadruped and biped robots. While MPC has demonstrated advantages in legged robots, existing studies often oversimplify the leg swing progress, relying merely on simple trajectory interpolation methods. This severely constrains the foot's environmental interaction capability, hindering tasks such as ball kicking. This study innovatively adapts the spatial-temporal trajectory planning method, which has been successful in drone applications, to bipedal robotic systems. The proposed approach autonomously generates foot trajectories that satisfy constraints on target kicking position, velocity, and acceleration while simultaneously optimizing swing phase duration. Experimental results demonstrate that the optimized trajectories closely mimic human kicking behavior, featuring a backswing motion. Simulation and hardware experiments confirm the algorithm's efficiency, with trajectory planning times under 1 ms, and its reliability, achieving nearly 100 % task completion accuracy when the soccer goal is within the range of -90° to 90°.
人形机器人足球面临若干挑战,特别是在进行激进的踢球动作时保持系统稳定性的同时实现精确的球轨迹控制。当前解决方案无论是基于位置的传统控制方法还是强化学习(RL)方法都表现出明显的局限性。模型预测控制(MPC)是普通四足和双足机器人的常用方法。虽然MPC在腿足机器人中展示了显著优势,但现有研究往往简化了腿部摆动过程,仅仅依靠简单的轨迹插值方法。这严重限制了脚部与环境的交互能力,阻碍了如踢球等任务的执行。 本研究创新性地将空间-时间轨迹规划方法(在无人机应用中已取得成功)应用于双足机器人系统。该方法能够自主生成满足目标踢球位置、速度和加速度约束的同时优化摆动阶段持续时间的脚部轨迹。实验结果显示,经过优化后的轨迹非常接近人类的踢球行为,并且包含了一个回摆动作。模拟和硬件实验证明了算法的有效性,轨迹规划时间低于1毫秒;同时,当足球目标在-90°到90°范围内时,任务完成准确率几乎达到100%。
https://arxiv.org/abs/2510.01843
With the rapid advancement of Large Language Models (LLMs), their capabilities in various automation domains, particularly Unmanned Aerial Vehicle (UAV) operations, have garnered increasing attention. Current research remains predominantly constrained to small-scale UAV applications, with most studies focusing on isolated components such as path planning for toy drones, while lacking comprehensive investigation of medium- and long-range UAV systems in real-world operational contexts. Larger UAV platforms introduce distinct challenges, including stringent requirements for airport-based take-off and landing procedures, adherence to complex regulatory frameworks, and specialized operational capabilities with elevated mission expectations. This position paper presents the Next-Generation LLM for UAV (NeLV) system -- a comprehensive demonstration and automation roadmap for integrating LLMs into multi-scale UAV operations. The NeLV system processes natural language instructions to orchestrate short-, medium-, and long-range UAV missions through five key technical components: (i) LLM-as-Parser for instruction interpretation, (ii) Route Planner for Points of Interest (POI) determination, (iii) Path Planner for waypoint generation, (iv) Control Platform for executable trajectory implementation, and (v) UAV monitoring. We demonstrate the system's feasibility through three representative use cases spanning different operational scales: multi-UAV patrol, multi-POI delivery, and multi-hop relocation. Beyond the current implementation, we establish a five-level automation taxonomy that charts the evolution from current LLM-as-Parser capabilities (Level 1) to fully autonomous LLM-as-Autopilot systems (Level 5), identifying technical prerequisites and research challenges at each stage.
https://arxiv.org/abs/2510.21739
This paper presents a robust neural control design for a three-drone slung payload transportation system to track a reference path under external disturbances. The control contraction metric (CCM) is used to generate a neural exponentially converging baseline controller while complying with control input saturation constraints. We also incorporate the uncertainty and disturbance estimator (UDE) technique to dynamically compensate for persistent disturbances. The proposed framework yields a modularized design, allowing the controller and estimator to perform their individual tasks and achieve a zero trajectory tracking error if the disturbances meet certain assumptions. The stability and robustness of the complete system, incorporating both the CCM controller and the UDE compensator, are presented. Simulations are conducted to demonstrate the capability of the proposed control design to follow complicated trajectories under external disturbances.
本文提出了一种针对三无人机吊载运输系统的鲁棒神经网络控制设计,该系统能够在外部干扰下追踪参考路径。利用控制收缩度量(CCM)生成一个在满足控制输入饱和约束的同时能够指数收敛的基准控制器。我们还引入了不确定性与扰动估计器(UDE)技术以动态补偿持续存在的扰动。所提出的框架实现了模块化设计,使得控制器和估计器各自独立完成任务,并且如果外部干扰符合一定假设条件,则可以实现零轨迹跟踪误差。文中还展示了结合了CCM控制器和UDE补偿器的整个系统的稳定性和鲁棒性。通过仿真验证了该控制设计方案在面对外界扰动时追踪复杂路径的能力。
https://arxiv.org/abs/2510.01489
After years of growth, drone-based delivery is transforming logistics. At its core, real-time 6-DoF drone pose tracking enables precise flight control and accurate drone landing. With the widespread availability of urban 3D maps, the Visual Positioning Service (VPS), a mobile pose estimation system, has been adapted to enhance drone pose tracking during the landing phase, as conventional systems like GPS are unreliable in urban environments due to signal attenuation and multi-path propagation. However, deploying the current VPS on drones faces limitations in both estimation accuracy and efficiency. In this work, we redesign drone-oriented VPS with the event camera and introduce EV-Pose to enable accurate, high-frequency 6-DoF pose tracking for accurate drone landing. EV-Pose introduces a spatio-temporal feature-instructed pose estimation module that extracts a temporal distance field to enable 3D point map matching for pose estimation; and a motion-aware hierarchical fusion and optimization scheme to enhance the above estimation in accuracy and efficiency, by utilizing drone motion in the \textit{early stage} of event filtering and the \textit{later stage} of pose optimization. Evaluation shows that EV-Pose achieves a rotation accuracy of 1.34$\degree$ and a translation accuracy of 6.9$mm$ with a tracking latency of 10.08$ms$, outperforming baselines by $>$50\%, \tmcrevise{thus enabling accurate drone landings.} Demo: this https URL
经过多年的成长,基于无人机的配送正在重塑物流行业。在核心层面,实时6自由度(6-DoF)无人机姿态跟踪使得精确飞行控制和准确着陆成为可能。随着城市3D地图的广泛普及,视觉定位服务(VPS),一个移动姿态估计系统,已被调整以增强无人机在降落阶段的姿态跟踪能力,在城市环境中传统的GPS等系统由于信号衰减和多路径传播而变得不可靠。 然而,在当前无人机上部署VPS面临准确性和效率上的限制。为了解决这个问题,我们在工作中重新设计了面向无人机的VPS,并引入了一种名为EV-Pose的技术,它利用事件相机实现了精确、高频次的6-DoF姿态跟踪,以实现精准无人机着陆。EV-Pose 引入了一个时空特征指导的姿态估计模块,该模块提取时间距离场,用于三维点地图匹配进行姿态估计;以及一个运动感知分层融合与优化方案,通过在事件过滤阶段利用无人机运动和在姿态优化阶段的后续应用来提升上述估计的准确性和效率。 评估结果表明,EV-Pose 在旋转精度方面达到了1.34°,平移精度为6.9毫米,并且跟踪延迟仅为10.08毫秒,相对于基线技术,性能提高了超过50%,从而实现了精确无人机着陆。演示视频可在此链接观看:[此 https URL]
https://arxiv.org/abs/2510.00646
Ecological monitoring is increasingly automated by vision models, yet opaque predictions limit trust and field adoption. We present an inpainting-guided, perturbation-based explanation technique that produces photorealistic, mask-localized edits that preserve scene context. Unlike masking or blurring, these edits stay in-distribution and reveal which fine-grained morphological cues drive predictions in tasks such as species recognition and trait attribution. We demonstrate the approach on a YOLOv9 detector fine-tuned for harbor seal detection in Glacier Bay drone imagery, using Segment-Anything-Model-refined masks to support two interventions: (i) object removal/replacement (e.g., replacing seals with plausible ice/water or boats) and (ii) background replacement with original animals composited onto new scenes. Explanations are assessed by re-scoring perturbed images (flip rate, confidence drop) and by expert review for ecological plausibility and interpretability. The resulting explanations localize diagnostic structures, avoid deletion artifacts common to traditional perturbations, and yield domain-relevant insights that support expert validation and more trustworthy deployment of AI in ecology.
生态监测越来越依赖于视觉模型的自动化,然而不透明的预测限制了信任和现场应用。我们提出了一种基于修复引导、扰动分析的解释技术,该技术可以生成保留场景上下文的真实感光编辑,并且这些编辑集中在特定的遮罩区域。与简单的蒙版或模糊处理不同,这种编辑保持在数据分布之内,并揭示了驱动预测的具体细粒度形态特征,在物种识别和属性归因等任务中尤为关键。 我们在冰川湾无人机图像上对YOLOv9检测器进行微调,用于海港 seal 的检测,并利用Segment-Anything-Model优化的掩码支持两种干预措施:(i)对象移除/替换(例如,将seal替换成合理的冰、水或船只),以及(ii)背景替换,即在新的场景中以原始动物为素材进行合成。我们通过重新评估扰动后的图像得分(翻转率和置信度下降)和专家评审的生态合理性及可解释性来评估这种方法的效果。 结果表明,这些解释能够定位诊断结构,避免传统扰动常见的删除伪影,并提供与领域相关的见解以支持专家验证和更值得信赖的人工智能在生态学中的应用部署。
https://arxiv.org/abs/2510.03317
Autonomous inspection systems are essential for ensuring the performance and longevity of industrial assets. Recently, agentic frameworks have demonstrated significant potential for automating inspection workflows but have been limited to digital tasks. Their application to physical assets in real-world environments, however, remains underexplored. In this work, our contributions are two-fold: first, we propose a hierarchical agentic framework for autonomous drone control, and second, a reasoning methodology for individual function executions which we refer to as ReActEval. Our framework focuses on visual inspection tasks in indoor industrial settings, such as interpreting industrial readouts or inspecting equipment. It employs a multi-agent system comprising a head agent and multiple worker agents, each controlling a single drone. The head agent performs high-level planning and evaluates outcomes, while worker agents implement ReActEval to reason over and execute low-level actions. Operating entirely in natural language, ReActEval follows a plan, reason, act, evaluate cycle, enabling drones to handle tasks ranging from simple navigation (e.g., flying forward 10 meters and land) to complex high-level tasks (e.g., locating and reading a pressure gauge). The evaluation phase serves as a feedback and/or replanning stage, ensuring actions align with user objectives while preventing undesirable outcomes. We evaluate the framework in a simulated environment with two worker agents, assessing performance qualitatively and quantitatively based on task completion across varying complexity levels and workflow efficiency. By leveraging natural language processing for agent communication, our approach offers a novel, flexible, and user-accessible alternative to traditional drone-based solutions, enabling autonomous problem-solving for industrial inspection without extensive user intervention.
自主检查系统对于确保工业资产的性能和寿命至关重要。最近,代理框架在自动化检查工作流程方面展示了巨大的潜力,但仅限于数字任务。然而,在现实世界环境中将其应用于物理资产的应用却鲜有探索。在这项工作中,我们的贡献包括两个部分:首先,我们提出了一种用于自主无人机控制的分层代理框架;其次,一种称为ReActEval的推理方法,用于个体功能执行。 本框架专注于室内工业环境中的视觉检查任务,例如解读工业读数或检查设备。它采用一个多代理系统,其中包括一个头代理人和多个工人代理人,每个代理人都控制着一架无人机。头代理人进行高层次计划并评估结果,而工人代理人则使用ReActEval来推理和执行低层次的动作。 ReActEval完全基于自然语言工作,遵循规划、推理、行动、评价的循环,使无人机能够处理从简单导航(例如向前飞行10米然后降落)到复杂的高层次任务(如定位和读取压力表)等各种任务。评估阶段作为反馈和/或重新计划的环节,确保动作符合用户目标,并防止不良结果的发生。 我们通过两个工人代理人在模拟环境中对框架进行评估,根据不同复杂度级别的任务完成情况以及工作流程效率从定性和定量两方面来评估其性能表现。 借助自然语言处理实现代理间通信,我们的方法为传统基于无人机的解决方案提供了一种新颖、灵活且用户友好的替代方案,它能够使工业检查中的自主问题解决成为可能,并且无需广泛的用户干预。
https://arxiv.org/abs/2510.00259
Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at this https URL
https://arxiv.org/abs/2510.00225
Embodied Artificial Intelligence (AI) promises to handle edge cases in robotic vehicle systems where data is scarce by using common-sense reasoning grounded in perception and action to generalize beyond training distributions and adapt to novel real-world situations. These capabilities, however, also create new security risks. In this paper, we introduce CHAI (Command Hijacking against embodied AI), a new class of prompt-based attacks that exploit the multimodal language interpretation abilities of Large Visual-Language Models (LVLMs). CHAI embeds deceptive natural language instructions, such as misleading signs, in visual input, systematically searches the token space, builds a dictionary of prompts, and guides an attacker model to generate Visual Attack Prompts. We evaluate CHAI on four LVLM agents; drone emergency landing, autonomous driving, and aerial object tracking, and on a real robotic vehicle. Our experiments show that CHAI consistently outperforms state-of-the-art attacks. By exploiting the semantic and multimodal reasoning strengths of next-generation embodied AI systems, CHAI underscores the urgent need for defenses that extend beyond traditional adversarial robustness.
嵌入式人工智能(AI)承诺通过使用基于感知和行动的常识推理来处理机器人车辆系统中数据稀缺的情况,从而能够超越训练分布并在新的现实世界情境下进行适应。然而,这些能力也带来了新的安全风险。本文介绍了 CHAI (针对嵌入式 AI 的命令劫持攻击),这是一种新型的基于提示的攻击方法,利用大型视觉-语言模型(LVLM)的多模态语言解释能力。CHAI 将欺骗性的自然语言指令,如误导性标志,嵌入到视觉输入中,并系统地搜索词元空间、构建提示字典,引导攻击者模型生成视觉攻击提示。 我们在四种 LVLM 代理上评估了 CHAI:无人机紧急着陆、自动驾驶和空中物体跟踪,以及在实际的机器人车辆上。我们的实验表明,CHAI 在面对最先进的攻击时始终表现出色。通过利用下一代嵌入式 AI 系统的语义和多模态推理能力,CHAI 强调了需要超越传统对抗鲁棒性的防御措施以应对安全挑战。
https://arxiv.org/abs/2510.00181
Autonomous drones must often respond to sudden events, such as alarms, faults, or unexpected changes in their environment, that require immediate and adaptive decision-making. Traditional approaches rely on safety engineers hand-coding large sets of recovery rules, but this strategy cannot anticipate the vast range of real-world contingencies and quickly becomes incomplete. Recent advances in embodied AI, powered by large visual language models, provide commonsense reasoning to assess context and generate appropriate actions in real time. We demonstrate this capability in a simulated urban benchmark in the Unreal Engine, where drones dynamically interpret their surroundings and decide on sudden maneuvers for safe landings. Our results show that embodied AI makes possible a new class of adaptive recovery and decision-making pipelines that were previously infeasible to design by hand, advancing resilience and safety in autonomous aerial systems.
自主无人机常常需要应对突然事件,如警报、故障或环境中的意外变化,这些情况要求即时和适应性的决策。传统方法依赖于安全工程师手动编写大量的恢复规则,但这种方法无法预测现实世界中广泛多样的突发状况,并且很快就会变得不完整。近年来,基于大型视觉语言模型的具身人工智能(Embodied AI)的进步提供了常识推理能力,能够评估环境并实时生成适当的行动。我们在虚幻引擎中的模拟城市基准测试中展示了这种能力,无人机可以动态地解释周围的环境并在需要时做出突然的机动以确保安全着陆。我们的研究结果表明,具身AI使得设计以前难以手动实现的新一代适应性恢复和决策流程成为可能,从而提升了自主空中系统的弹性和安全性。
https://arxiv.org/abs/2510.00167