Paper Reading AI Learner

SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learning

2025-10-16 15:16:26
Aderik Verraest, Stavrow Bahnam, Robin Ferede, Guido de Croon, Christophe De Wagter

Abstract

Autonomous drone racing (ADR) systems have recently achieved champion-level performance, yet remain highly specific to drone racing. While end-to-end vision-based methods promise broader applicability, no system to date simultaneously achieves full sim-to-real transfer, onboard execution, and champion-level performance. In this work, we present SkyDreamer, to the best of our knowledge, the first end-to-end vision-based ADR policy that maps directly from pixel-level representations to motor commands. SkyDreamer builds on informed Dreamer, a model-based reinforcement learning approach where the world model decodes to privileged information only available during training. By extending this concept to end-to-end vision-based ADR, the world model effectively functions as an implicit state and parameter estimator, greatly improving interpretability. SkyDreamer runs fully onboard without external aid, resolves visual ambiguities by tracking progress using the state decoded from the world model's hidden state, and requires no extrinsic camera calibration, enabling rapid deployment across different drones without retraining. Real-world experiments show that SkyDreamer achieves robust, high-speed flight, executing tight maneuvers such as an inverted loop, a split-S and a ladder, reaching speeds of up to 21 m/s and accelerations of up to 6 g. It further demonstrates a non-trivial visual sim-to-real transfer by operating on poor-quality segmentation masks, and exhibits robustness to battery depletion by accurately estimating the maximum attainable motor RPM and adjusting its flight path in real-time. These results highlight SkyDreamer's adaptability to important aspects of the reality gap, bringing robustness while still achieving extremely high-speed, agile flight.

Abstract (translated)

最近,自主无人机竞速(ADR)系统已经达到了冠军级别的表现,但这些系统仍高度特定于无人机竞速领域。虽然端到端视觉方法承诺更广泛的适用性,但迄今为止还没有一个系统能够同时实现从仿真环境到真实环境的完全迁移、在机上执行以及达到冠军级别性能。在这项工作中,我们介绍了SkyDreamer,在我们的知识范围内,它是第一个直接将像素级表示映射为电机指令的基于端到端视觉的ADR策略。SkyDreamer建立在知情梦想家的基础上,这是一种模型强化学习方法,其中世界模型解码成仅在训练期间可用的特权信息。通过将这一概念扩展到基于端到端视觉的ADR中,世界模型实际上充当了隐式状态和参数估计器,大大提高了可解释性。 SkyDreamer完全自主运行,无需外部支持,在飞行过程中利用从世界模型隐藏状态解码出的状态来解决视觉模糊问题,并且不需要外在相机校准,这使得它能够在不同无人机上快速部署而无需重新训练。真实世界的实验表明,SkyDreamer实现了稳健的高速飞行,执行诸如倒立环、分叉S和梯形等紧密机动动作,最高速度可达21米/秒,最大加速度可达6g。此外,通过在质量较差的分割掩码上操作并实时准确估计可达到的最大电机转速,并调整其飞行路径,SkyDreamer还展示了非平凡的视觉仿真到现实转移能力以及对电池耗尽的鲁棒性。 这些结果突显了SkyDreamer适应真实性和模拟环境之间重要差距方面的能力,在保持极高速度和敏捷飞行的同时实现稳健性能。

URL

https://arxiv.org/abs/2510.14783

PDF

https://arxiv.org/pdf/2510.14783.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot