Abstract
Immediate damage assessment is essential after natural catastrophes; yet, conventional hand evaluation techniques are sluggish and perilous. Although satellite and unmanned aerial vehicle (UAV) photos offer extensive perspectives of impacted regions, current computer vision methodologies generally yield just classification labels or segmentation masks, so constraining their capacity to deliver a thorough situational comprehension. We introduce the Vision Language Caption Enhancer (VLCE), a multimodal system designed to produce comprehensive, contextually-informed explanations of disaster imagery. VLCE employs a dual-architecture approach: a CNN-LSTM model with a ResNet50 backbone pretrained on EuroSat satellite imagery for the xBD dataset, and a Vision Transformer (ViT) model pretrained on UAV pictures for the RescueNet dataset. Both systems utilize external semantic knowledge from ConceptNet and WordNet to expand vocabulary coverage and improve description accuracy. We assess VLCE in comparison to leading vision-language models (LLaVA and QwenVL) utilizing CLIPScore for semantic alignment and InfoMetIC for caption informativeness. Experimental findings indicate that VLCE markedly surpasses baseline models, attaining a maximum of 95.33% on InfoMetIC while preserving competitive semantic alignment. Our dual-architecture system demonstrates significant potential for improving disaster damage assessment by automating the production of actionable, information-dense descriptions from satellite and drone photos.
Abstract (translated)
自然灾害发生后,立即进行损害评估至关重要;然而,传统的手工评估技术既缓慢又危险。虽然卫星和无人飞行器(UAV)拍摄的照片可以提供受影响区域的广泛视角,但当前的计算机视觉方法通常只能生成分类标签或分割掩模,这限制了它们提供全面情况理解的能力。我们引入了视觉语言描述增强器(VLCE),这是一个多模式系统,旨在产生对灾难图像的详尽、语境相关的解释。 VLCE采用双架构设计:对于xBD数据集,使用在EuroSat卫星图像上预训练的CNN-LSTM模型与ResNet50骨干网络;对于RescueNet数据集,则使用了在UAV图片上预训练的Vision Transformer(ViT)模型。这两个系统都利用ConceptNet和WordNet等外部语义知识来扩展词汇覆盖率并提高描述准确性。 我们通过CLIPScore评估VLCE与其他领先的视觉-语言模型(LLaVA和QwenVL)之间的语义对齐,并使用InfoMetIC评估标题信息的丰富程度。实验结果表明,VLCE显著超越了基准模型,在InfoMetIC上达到了最高95.33%的同时保持了竞争性的语义对齐度。 我们的双架构系统在从卫星和无人机照片中自动生成具有行动力的信息密集型描述方面展示了巨大的潜力,从而提高灾害损害评估的效率。
URL
https://arxiv.org/abs/2509.21609