Paper Reading AI Learner

SINGER: An Onboard Generalist Vision-Language Navigation Policy for Drones

2025-09-23 03:57:34
Maximilian Adang, JunEn Low, Ola Shorinwa, Mac Schwager

Abstract

Large vision-language models have driven remarkable progress in open-vocabulary robot policies, e.g., generalist robot manipulation policies, that enable robots to complete complex tasks specified in natural language. Despite these successes, open-vocabulary autonomous drone navigation remains an unsolved challenge due to the scarcity of large-scale demonstrations, real-time control demands of drones for stabilization, and lack of reliable external pose estimation modules. In this work, we present SINGER for language-guided autonomous drone navigation in the open world using only onboard sensing and compute. To train robust, open-vocabulary navigation policies, SINGER leverages three central components: (i) a photorealistic language-embedded flight simulator with minimal sim-to-real gap using Gaussian Splatting for efficient data generation, (ii) an RRT-inspired multi-trajectory generation expert for collision-free navigation demonstrations, and these are used to train (iii) a lightweight end-to-end visuomotor policy for real-time closed-loop control. Through extensive hardware flight experiments, we demonstrate superior zero-shot sim-to-real transfer of our policy to unseen environments and unseen language-conditioned goal objects. When trained on ~700k-1M observation action pairs of language conditioned visuomotor data and deployed on hardware, SINGER outperforms a velocity-controlled semantic guidance baseline by reaching the query 23.33% more on average, and maintains the query in the field of view 16.67% more on average, with 10% fewer collisions.

Abstract (translated)

大型视觉-语言模型在开放式词汇机器人策略方面取得了显著进展,例如通用型机器人操作策略,使机器人能够完成用自然语言指定的复杂任务。尽管取得这些成就,但由于缺乏大规模演示、无人机实时控制的需求以及可靠的外部姿态估计模块不足等原因,开放词汇自主无人机导航仍是一项未解决的挑战。本文介绍了SINGER系统,该系统利用机载传感器和计算能力,在开放世界中实现由语言引导的自主无人机导航。 为了训练稳健的开放式词汇导航策略,SINGER采用了三个核心组件:(i)一个基于高斯点播技术进行高效数据生成、具有最小模拟到真实差距的逼真飞行仿真器;(ii)一种受RRT启发的多轨迹生成专家系统,用于提供无碰撞导航演示。这些被用来训练(iii)一个轻量级端到端视觉运动策略,以实现实时闭环控制。 通过大量的硬件飞行实验,我们证明了我们的政策在未见过的环境和语言条件下的零样本模拟到真实转移具有优越性,并且当在约70万至100万个观察动作对的语言条件下训练后,在部署于实际设备上时,SINGER的表现超过了速度控制的语义引导基线方法:平均而言,前者达到查询目标的概率比后者高出23.33%,保持查询对象在视野中的概率高出了16.67%;并且碰撞次数减少了10%。

URL

https://arxiv.org/abs/2509.18610

PDF

https://arxiv.org/pdf/2509.18610.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot