Paper Reading AI Learner

GRaD-Nav++: Vision-Language Model Enabled Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamics

2025-06-16 21:12:27
Qianzhong Chen, Naixiang Gao, Suning Huang, JunEn Low, Timothy Chen, Jiankai Sun, Mac Schwager

Abstract

Autonomous drones capable of interpreting and executing high-level language instructions in unstructured environments remain a long-standing goal. Yet existing approaches are constrained by their dependence on hand-crafted skills, extensive parameter tuning, or computationally intensive models unsuitable for onboard use. We introduce GRaD-Nav++, a lightweight Vision-Language-Action (VLA) framework that runs fully onboard and follows natural-language commands in real time. Our policy is trained in a photorealistic 3D Gaussian Splatting (3DGS) simulator via Differentiable Reinforcement Learning (DiffRL), enabling efficient learning of low-level control from visual and linguistic inputs. At its core is a Mixture-of-Experts (MoE) action head, which adaptively routes computation to improve generalization while mitigating forgetting. In multi-task generalization experiments, GRaD-Nav++ achieves a success rate of 83% on trained tasks and 75% on unseen tasks in simulation. When deployed on real hardware, it attains 67% success on trained tasks and 50% on unseen ones. In multi-environment adaptation experiments, GRaD-Nav++ achieves an average success rate of 81% across diverse simulated environments and 67% across varied real-world settings. These results establish a new benchmark for fully onboard Vision-Language-Action (VLA) flight and demonstrate that compact, efficient models can enable reliable, language-guided navigation without relying on external infrastructure.

Abstract (translated)

自主无人机能够解读并执行在无结构环境中下达的高级语言指令,一直是长久以来的目标。然而,现有的方法受限于手工设计技能、大量参数调整或计算密集型模型的使用,后者不适合机载部署。我们引入了GRaD-Nav++,这是一个轻量级的视觉-语言-行动(VLA)框架,能够在无人机上完全独立运行,并实时遵循自然语言指令。 我们的策略通过可微强化学习(DiffRL)在逼真的3D高斯点集(3DGS)模拟器中进行训练,从视觉和语言输入中高效地学习低级控制。核心部分是一个专家混合模型(MoE),该模型能够自适应地分配计算资源以提高泛化能力,并减少遗忘效应。 在多任务泛化实验中,GRaD-Nav++在仿真环境中针对已训练的任务达到了83%的成功率,对于未见过的任务为75%。当部署到真实硬件上时,它对已训练的任务的成功率为67%,而面对未知环境下的任务成功率则为50%。 在多环境适应实验中,GRaD-Nav++在多样化的仿真环境中平均成功率达到81%,而在不同的现实世界设置下则达到了67%。这些结果为完全机载的视觉语言行动飞行设定了新的基准,并证明了紧凑且高效的模型能够在无需依赖外部基础设施的情况下实现可靠的语言引导导航。

URL

https://arxiv.org/abs/2506.14009

PDF

https://arxiv.org/pdf/2506.14009.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot