Paper Reading AI Learner

UAV4D: Dynamic Neural Rendering of Human-Centric UAV Imagery using Gaussian Splatting

2025-06-05 13:21:09
Jaehoon Choi, Dongki Jung, Christopher Maxey, Yonghan Lee, Sungmin Eum, Dinesh Manocha, Heesung Kwon

Abstract

Despite significant advancements in dynamic neural rendering, existing methods fail to address the unique challenges posed by UAV-captured scenarios, particularly those involving monocular camera setups, top-down perspective, and multiple small, moving humans, which are not adequately represented in existing datasets. In this work, we introduce UAV4D, a framework for enabling photorealistic rendering for dynamic real-world scenes captured by UAVs. Specifically, we address the challenge of reconstructing dynamic scenes with multiple moving pedestrians from monocular video data without the need for additional sensors. We use a combination of a 3D foundation model and a human mesh reconstruction model to reconstruct both the scene background and humans. We propose a novel approach to resolve the scene scale ambiguity and place both humans and the scene in world coordinates by identifying human-scene contact points. Additionally, we exploit the SMPL model and background mesh to initialize Gaussian splats, enabling holistic scene rendering. We evaluated our method on three complex UAV-captured datasets: VisDrone, Manipal-UAV, and Okutama-Action, each with distinct characteristics and 10~50 humans. Our results demonstrate the benefits of our approach over existing methods in novel view synthesis, achieving a 1.5 dB PSNR improvement and superior visual sharpness.

Abstract (translated)

尽管动态神经渲染技术取得了显著进展,但现有方法未能解决无人机(UAV)捕捉场景中的独特挑战,特别是单目相机设置、顶部视图视角以及多个小型移动人物的问题,这些问题在现有的数据集中并未得到充分表示。在此工作中,我们引入了UAV4D框架,该框架旨在实现由无人机捕获的真实世界动态场景的逼真渲染。 具体而言,我们解决了从单目视频数据中重建包含多个人行者的动态场景的挑战,并且无需额外传感器的支持就能完成这一任务。为此,我们将3D基础模型与人体网格重建模型相结合,用于同时重建场景背景和人物。我们提出了一种新的方法来解决场景尺度模糊问题,并通过识别人物-场景接触点将两者置于世界坐标系中。此外,利用SMPL(Skeletal Modeling and Parameterization)模型以及背景网格初始化高斯点集,这使整个场景的渲染成为可能。 我们在三个复杂的无人机捕捉数据集上评估了我们的方法:VisDrone、Manipal-UAV和Okutama-Action,每个数据集中包含10至50个个体,并且各具特色。实验结果表明,与现有方法相比,我们的方法在新颖视图合成方面表现出显著优势,实现了高达1.5 dB的PSNR(峰值信噪比)改进以及更佳的视觉清晰度。

URL

https://arxiv.org/abs/2506.05011

PDF

https://arxiv.org/pdf/2506.05011.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot