Abstract
In this study, we present an end-to-end pipeline capable of converting drone-captured video streams into high-fidelity 3D reconstructions with minimal latency. Unmanned aerial vehicles (UAVs) are extensively used in aerial real-time perception applications. Moreover, recent advances in 3D Gaussian Splatting (3DGS) have demonstrated significant potential for real-time neural rendering. However, their integration into end-to-end UAV-based reconstruction and visualization systems remains underexplored. Our goal is to propose an efficient architecture that combines live video acquisition via RTMP streaming, synchronized sensor fusion, camera pose estimation, and 3DGS optimization, achieving continuous model updates and low-latency deployment within interactive visualization environments that supports immersive augmented and virtual reality (AR/VR) applications. Experimental results demonstrate that the proposed method achieves competitive visual fidelity, while delivering significantly higher rendering performance and substantially reduced end-to-end latency, compared to NeRF-based approaches. Reconstruction quality remains within 4-7\% of high-fidelity offline references, confirming the suitability of the proposed system for real-time, scalable augmented perception from aerial platforms.
Abstract (translated)
在这项研究中，我们提出了一种端到端的流水线，能够将无人机捕捉的视频流转换为高质量的3D重建模型，并且延迟极低。无人驾驶飞行器（UAV）在实时空中感知应用中被广泛使用。此外，最近关于3D高斯点阵（3D Gaussian Splatting, 3DGS）的进步已经展示了其实时神经渲染的巨大潜力。然而，它们集成到基于无人机的端到端重建和可视化系统中的方法仍需进一步探索。 我们的目标是提出一种高效的架构，该架构结合了通过RTMP流进行实时视频采集、传感器同步融合、摄像机姿态估计以及3DGS优化，从而实现持续模型更新，并在支持沉浸式增强现实（AR）和虚拟现实（VR）应用的交互可视化环境中部署低延迟。实验结果表明，所提出的方法实现了竞争性的视觉保真度，同时提供了比NeRF方法显著更高的渲染性能和大幅减少的端到端延迟。 重建质量与高质量离线参考之间的差距维持在4-7%，这证明了该系统适用于从空中平台进行实时、可扩展的增强感知。
URL
https://arxiv.org/abs/2602.20342
