Paper Reading AI Learner

Learning Real-World Acrobatic Flight from Human Preferences

2025-08-26 08:56:53
Colin Merk, Ismail Geles, Jiaxu Xing, Angel Romero, Giorgia Ramponi, Davide Scaramuzza

Abstract

Preference-based reinforcement learning (PbRL) enables agents to learn control policies without requiring manually designed reward functions, making it well-suited for tasks where objectives are difficult to formalize or inherently subjective. Acrobatic flight poses a particularly challenging problem due to its complex dynamics, rapid movements, and the importance of precise execution. In this work, we explore the use of PbRL for agile drone control, focusing on the execution of dynamic maneuvers such as powerloops. Building on Preference-based Proximal Policy Optimization (Preference PPO), we propose Reward Ensemble under Confidence (REC), an extension to the reward learning objective that improves preference modeling and learning stability. Our method achieves 88.4% of the shaped reward performance, compared to 55.2% with standard Preference PPO. We train policies in simulation and successfully transfer them to real-world drones, demonstrating multiple acrobatic maneuvers where human preferences emphasize stylistic qualities of motion. Furthermore, we demonstrate the applicability of our probabilistic reward model in a representative MuJoCo environment for continuous control. Finally, we highlight the limitations of manually designed rewards, observing only 60.7% agreement with human preferences. These results underscore the effectiveness of PbRL in capturing complex, human-centered objectives across both physical and simulated domains.

Abstract (translated)

基于偏好的强化学习(PbRL)使智能体能够在无需手动设计奖励函数的情况下学习控制策略,因此非常适合目标难以形式化或本质上具有主观性的任务。特技飞行因其复杂的动力学特性、快速的运动以及执行精确度的重要性而构成一个特别具挑战性的问题。在这项工作中,我们探讨了将PbRL用于敏捷无人机控制的可能性,并专注于动态机动动作(如螺旋筋斗)的执行。在基于偏好的近端策略优化(Preference PPO)的基础上,我们提出了置信度下的奖励集成(REC),这是一种对奖励学习目标进行扩展的方法,能够改进偏好建模和学习稳定性。我们的方法实现了88.4%的塑形奖励性能,而标准的Preference PPO仅实现55.2%。 我们在模拟环境中训练策略,并成功将其转移到真实的无人机上,展示了多个强调运动风格质量的人类偏好的特技动作。此外,我们还在一个具有代表性的MuJoCo环境中演示了我们的概率性奖励模型在连续控制任务中的适用性。最后,我们指出了手动设计奖励的局限性,观察到仅达到60.7%与人类偏好一致的程度。 这些结果强调了PbRL在捕捉复杂且以人为中心的目标方面的有效性,无论是在物理环境还是模拟环境中都是如此。

URL

https://arxiv.org/abs/2508.18817

PDF

https://arxiv.org/pdf/2508.18817.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot