Abstract
Inspecting confined industrial infrastructure, such as ventilation shafts, is a hazardous and inefficient task for humans. Unmanned Aerial Vehicles (UAVs) offer a promising alternative, but GPS-denied environments require robust control policies to prevent collisions. Deep Reinforcement Learning (DRL) has emerged as a powerful framework for developing such policies, and this paper provides a comparative study of two leading DRL algorithms for this task: the on-policy Proximal Policy Optimization (PPO) and the off-policy Soft Actor-Critic (SAC). The training was conducted with procedurally generated duct environments in Genesis simulation environment. A reward function was designed to guide a drone through a series of waypoints while applying a significant penalty for collisions. PPO learned a stable policy that completed all evaluation episodes without collision, producing smooth trajectories. By contrast, SAC consistently converged to a suboptimal behavior that traversed only the initial segments before failure. These results suggest that, in hazard-dense navigation, the training stability of on-policy methods can outweigh the nominal sample efficiency of off-policy algorithms. More broadly, the study provides evidence that procedurally generated, high-fidelity simulations are effective testbeds for developing and benchmarking robust navigation policies.
Abstract (translated)
检查工业设施内部的受限区域,如通风井等,对于人类来说是一项危险且低效的任务。无人飞行器(UAV)提供了一种有前景的替代方案,但在无GPS信号的环境中操作时,需要具备强大的控制策略以防止碰撞事故的发生。深度强化学习(DRL)已成为开发此类控制策略的强大框架,并且本文通过对比研究两种领先于该任务领域的DRL算法——基于策略的近端策略优化(PPO)和非基于策略的软演员-评论家方法(SAC),来展示其效用。 训练过程在Genesis仿真环境中使用程序生成的通风管道环境进行。奖励函数被设计用来引导无人机按照一系列航点飞行,并对碰撞施加重罚。实验结果显示,PPO算法能够学习出一个稳定性的策略,在所有评估阶段均能无碰撞地完成任务,产生平稳的轨迹;而SAC则始终收敛到一种次优行为模式,即只通过最初的一些路段后便无法继续。 这些结果表明,在高风险密度环境中导航时,基于策略方法训练过程中的稳定性可能比非基于策略算法名义上的样本效率更为重要。更广泛地说,这项研究提供了证据,证明程序生成的、高度真实的仿真环境是开发和测试稳健导航策略的有效平台。
URL
https://arxiv.org/abs/2508.16807