Abstract
As aerial platforms evolve from passive observers to active manipulators, the challenge shifts toward designing intuitive interfaces that allow non-expert users to command these systems naturally. This work introduces a novel concept of autonomous aerial manipulation system capable of interpreting high-level natural language commands to retrieve objects and deliver them to a human user. The system is intended to integrate a MediaPipe based on Grounding DINO and a Vision-Language-Action (VLA) model with a custom-built drone equipped with a 1-DOF gripper and an Intel RealSense RGB-D camera. VLA performs semantic reasoning to interpret the intent of a user prompt and generates a prioritized task queue for grasping of relevant objects in the scene. Grounding DINO and dynamic A* planning algorithm are used to navigate and safely relocate the object. To ensure safe and natural interaction during the handover phase, the system employs a human-centric controller driven by MediaPipe. This module provides real-time human pose estimation, allowing the drone to employ visual servoing to maintain a stable, distinct position directly in front of the user, facilitating a comfortable handover. We demonstrate the system's efficacy through real-world experiments for localization and navigation, which resulted in a 0.164m, 0.070m, and 0.084m of max, mean euclidean, and root-mean squared errors, respectively, highlighting the feasibility of VLA for aerial manipulation operations.
Abstract (translated)
随着空中平台从被动观察者发展为主动操作者，设计直观的用户界面成为挑战，以使非专家用户能够自然地控制这些系统。这项工作提出了一种新颖的概念——自主空中操控系统，该系统能理解高级别的自然语言指令来检索物体并将其交付给人类用户。这个系统整合了基于Grounding DINO和MediaPipe的Vision-Language-Action (VLA)模型以及一个配备了1-DOF夹具和Intel RealSense RGB-D相机的定制无人机。 VLA模型执行语义推理，以理解用户的意图，并生成优先级任务队列来抓取场景中的相关物体。使用Grounding DINO和动态A*规划算法进行导航并安全地重新定位对象。为了确保在交接阶段的安全且自然互动，该系统采用了由MediaPipe驱动的人体中心控制器模块。这个模块提供实时人体姿态估计功能，使无人机能够采用视觉伺服技术来保持一个稳定、明显的姿态，直接位于用户前方，从而实现舒适的物体交付。 通过实际实验验证了系统的有效性，在定位和导航方面取得了最大误差为0.164米，平均欧氏距离误差为0.070米，均方根误差为0.084米的成绩。这些结果突显了VLA模型在空中操控操作中的可行性。
URL
https://arxiv.org/abs/2601.13809
