Abstract
Autonomous drones capable of interpreting and executing high-level language instructions in unstructured environments remain a long-standing goal. Yet existing approaches are constrained by their dependence on hand-crafted skills, extensive parameter tuning, or computationally intensive models unsuitable for onboard use. We introduce GRaD-Nav++, a lightweight Vision-Language-Action (VLA) framework that runs fully onboard and follows natural-language commands in real time. Our policy is trained in a photorealistic 3D Gaussian Splatting (3DGS) simulator via Differentiable Reinforcement Learning (DiffRL), enabling efficient learning of low-level control from visual and linguistic inputs. At its core is a Mixture-of-Experts (MoE) action head, which adaptively routes computation to improve generalization while mitigating forgetting. In multi-task generalization experiments, GRaD-Nav++ achieves a success rate of 83% on trained tasks and 75% on unseen tasks in simulation. When deployed on real hardware, it attains 67% success on trained tasks and 50% on unseen ones. In multi-environment adaptation experiments, GRaD-Nav++ achieves an average success rate of 81% across diverse simulated environments and 67% across varied real-world settings. These results establish a new benchmark for fully onboard Vision-Language-Action (VLA) flight and demonstrate that compact, efficient models can enable reliable, language-guided navigation without relying on external infrastructure.
Abstract (translated)
自主无人机能够解读并执行在无结构环境中下达的高级语言指令,一直是长久以来的目标。然而,现有的方法受限于手工设计技能、大量参数调整或计算密集型模型的使用,后者不适合机载部署。我们引入了GRaD-Nav++,这是一个轻量级的视觉-语言-行动(VLA)框架,能够在无人机上完全独立运行,并实时遵循自然语言指令。 我们的策略通过可微强化学习(DiffRL)在逼真的3D高斯点集(3DGS)模拟器中进行训练,从视觉和语言输入中高效地学习低级控制。核心部分是一个专家混合模型(MoE),该模型能够自适应地分配计算资源以提高泛化能力,并减少遗忘效应。 在多任务泛化实验中,GRaD-Nav++在仿真环境中针对已训练的任务达到了83%的成功率,对于未见过的任务为75%。当部署到真实硬件上时,它对已训练的任务的成功率为67%,而面对未知环境下的任务成功率则为50%。 在多环境适应实验中,GRaD-Nav++在多样化的仿真环境中平均成功率达到81%,而在不同的现实世界设置下则达到了67%。这些结果为完全机载的视觉语言行动飞行设定了新的基准,并证明了紧凑且高效的模型能够在无需依赖外部基础设施的情况下实现可靠的语言引导导航。
URL
https://arxiv.org/abs/2506.14009