Abstract
Drones equipped with overhead manipulators offer unique capabilities for inspection, maintenance, and contact-based interaction. However, the motion of the drone and its manipulator is tightly linked, and even small attitude changes caused by wind or control imperfections shift the end-effector away from its intended path. This coupling makes reliable tracking difficult and also limits the direct use of learning-based arm controllers that were originally designed for fixed-base robots. These effects appear consistently in our tests whenever the UAV body experiences drift or rapid attitude corrections. To address this behavior, we develop a reinforcement-learning (RL) framework with a transformer-based double deep Q learning (DDQN), with the core idea of using an adaptive beam-search planner that applies a short-horizon beam search over candidate control sequences using the learned critic as the forward estimator. This allows the controller to anticipate the end-effector's motion through simulated rollouts rather than executing those actions directly on the actual model, realizing a software-in-the-loop (SITL) approach. The lookahead relies on value estimates from a Transformer critic that processes short sequences of states, while a DDQN backbone provides the one-step targets needed to keep the learning process stable. Evaluated on a 3-DoF aerial manipulator under identical training conditions, the proposed meta-adaptive planner shows the strongest overall performance with a 10.2% reward increase, a substantial reduction in mean tracking error (from about 6% to 3%), and a 29.6% improvement in the combined reward-error metric relative to the DDQN baseline. Our method exhibits elevated stability in tracking target tip trajectory (by maintaining 5 cm tracking error) when the drone base exhibits drifts due to external disturbances, as opposed to the fixed-beam and Transformer-only variants.
Abstract (translated)
配备顶部机械臂的无人机在检测、维护及接触式交互方面具有独特能力。然而，无人机本体与机械臂的运动紧密耦合，即便由风况或控制偏差引起的微小姿态变化，也会导致末端执行器偏离预定轨迹。这种耦合不仅使可靠跟踪变得困难，也限制了最初为固定基座机器人设计的学习型机械臂控制器的直接应用。在我们的测试中，每当无人机本体出现漂移或快速姿态修正时，上述效应便会持续显现。为应对此问题，我们开发了一种基于Transformer的双深度Q学习（DDQN）强化学习框架，其核心思想是采用自适应波束搜索规划器：利用已学习的评判网络作为前向估计器，对候选控制序列进行短时域波束搜索。这使控制器能够通过模拟推演而非在实际模型上直接执行来预判末端执行器运动，从而实现软件在环（SITL）方法。该前瞻机制依赖于处理状态短序列的Transformer评判网络提供的价值估计，而DDQN主干网络则提供维持学习过程稳定所需的一步目标。在相同训练条件下对三自由度空中机械臂进行评估，所提出的元自适应规划器展现出最优综合性能：奖励提升10.2%，平均跟踪误差显著降低（从约6%降至3%），且相较于DDQN基线，综合奖励-误差指标改善29.6%。当无人机基座因外部扰动产生漂移时，本方法在跟踪末端轨迹方面表现出更高稳定性（可将跟踪误差维持在5厘米内），优于固定波束及纯Transformer变体方案。
URL
https://arxiv.org/abs/2603.26612
