Abstract
Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf{\underline{SVM-City}}, deriving from multi\textbf{\underline{S}}cale scenarios with multi\textbf{\underline{V}}iew and multi\textbf{\underline{M}}odal instruction tuning data. It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf{\underline{City-VLM}}. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves $18.14 \%$ performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes.
Abstract (translated)
场景理解使智能代理能够解释和理解其环境。尽管现有的大型视觉语言模型(LVLM)在室内家居任务的场景理解方面已经取得了进展,但当应用于户外大规模场景理解时,它们面临着两个重要的局限性。首先,室外情景通常包括通过多种传感器从多个视角观察到的大规模环境(如鸟瞰图和地面视图),而现有的室内LVLM主要分析单个视觉模态,在人类视角下的建筑物尺度背景下进行研究。其次,现有的LVLM缺乏多领域感知户外数据,并且难以有效整合2D和3D视觉信息。 为了克服上述限制,我们建立了第一个多领域感知的户外场景理解数据集,名为**SVM-City**,源自多种规模的情景、多视角和多模态指令调优数据。该数据集中含有420,000张图像和4,811M点云,并从车辆、低空无人机、高空飞机及卫星采集了567,000个问题-答案对。 为了有效融合缺失单一模态情况下的多模式数据,我们引入了一种不完整的多模态学习方法来建模户外场景理解,并设计了一个名为**City-VLM**的LVLM。通过构建联合概率分布空间而非直接进行显式融合操作(如串联),实现了多模式融合。 在三个典型的户外场景理解任务上的实验结果显示,City-VLM在问答任务上平均超越现有LVLM18.14%的性能表现。我们的方法在多个户外场景中展示了实用性和泛化能力。
URL
https://arxiv.org/abs/2507.12795