Abstract
Despite significant advancements in dynamic neural rendering, existing methods fail to address the unique challenges posed by UAV-captured scenarios, particularly those involving monocular camera setups, top-down perspective, and multiple small, moving humans, which are not adequately represented in existing datasets. In this work, we introduce UAV4D, a framework for enabling photorealistic rendering for dynamic real-world scenes captured by UAVs. Specifically, we address the challenge of reconstructing dynamic scenes with multiple moving pedestrians from monocular video data without the need for additional sensors. We use a combination of a 3D foundation model and a human mesh reconstruction model to reconstruct both the scene background and humans. We propose a novel approach to resolve the scene scale ambiguity and place both humans and the scene in world coordinates by identifying human-scene contact points. Additionally, we exploit the SMPL model and background mesh to initialize Gaussian splats, enabling holistic scene rendering. We evaluated our method on three complex UAV-captured datasets: VisDrone, Manipal-UAV, and Okutama-Action, each with distinct characteristics and 10~50 humans. Our results demonstrate the benefits of our approach over existing methods in novel view synthesis, achieving a 1.5 dB PSNR improvement and superior visual sharpness.
Abstract (translated)
尽管动态神经渲染技术取得了显著进展,但现有方法未能解决无人机(UAV)捕捉场景中的独特挑战,特别是单目相机设置、顶部视图视角以及多个小型移动人物的问题,这些问题在现有的数据集中并未得到充分表示。在此工作中,我们引入了UAV4D框架,该框架旨在实现由无人机捕获的真实世界动态场景的逼真渲染。 具体而言,我们解决了从单目视频数据中重建包含多个人行者的动态场景的挑战,并且无需额外传感器的支持就能完成这一任务。为此,我们将3D基础模型与人体网格重建模型相结合,用于同时重建场景背景和人物。我们提出了一种新的方法来解决场景尺度模糊问题,并通过识别人物-场景接触点将两者置于世界坐标系中。此外,利用SMPL(Skeletal Modeling and Parameterization)模型以及背景网格初始化高斯点集,这使整个场景的渲染成为可能。 我们在三个复杂的无人机捕捉数据集上评估了我们的方法:VisDrone、Manipal-UAV和Okutama-Action,每个数据集中包含10至50个个体,并且各具特色。实验结果表明,与现有方法相比,我们的方法在新颖视图合成方面表现出显著优势,实现了高达1.5 dB的PSNR(峰值信噪比)改进以及更佳的视觉清晰度。
URL
https://arxiv.org/abs/2506.05011