Paper Reading AI Learner

VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment

2025-09-25 21:21:00
Md. Mahfuzur Rahman, Kishor Datta Gupta, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Roy George

Abstract

Immediate damage assessment is essential after natural catastrophes; yet, conventional hand evaluation techniques are sluggish and perilous. Although satellite and unmanned aerial vehicle (UAV) photos offer extensive perspectives of impacted regions, current computer vision methodologies generally yield just classification labels or segmentation masks, so constraining their capacity to deliver a thorough situational comprehension. We introduce the Vision Language Caption Enhancer (VLCE), a multimodal system designed to produce comprehensive, contextually-informed explanations of disaster imagery. VLCE employs a dual-architecture approach: a CNN-LSTM model with a ResNet50 backbone pretrained on EuroSat satellite imagery for the xBD dataset, and a Vision Transformer (ViT) model pretrained on UAV pictures for the RescueNet dataset. Both systems utilize external semantic knowledge from ConceptNet and WordNet to expand vocabulary coverage and improve description accuracy. We assess VLCE in comparison to leading vision-language models (LLaVA and QwenVL) utilizing CLIPScore for semantic alignment and InfoMetIC for caption informativeness. Experimental findings indicate that VLCE markedly surpasses baseline models, attaining a maximum of 95.33% on InfoMetIC while preserving competitive semantic alignment. Our dual-architecture system demonstrates significant potential for improving disaster damage assessment by automating the production of actionable, information-dense descriptions from satellite and drone photos.

Abstract (translated)

自然灾害发生后,立即进行损害评估至关重要;然而,传统的手工评估技术既缓慢又危险。虽然卫星和无人飞行器(UAV)拍摄的照片可以提供受影响区域的广泛视角,但当前的计算机视觉方法通常只能生成分类标签或分割掩模,这限制了它们提供全面情况理解的能力。我们引入了视觉语言描述增强器(VLCE),这是一个多模式系统,旨在产生对灾难图像的详尽、语境相关的解释。 VLCE采用双架构设计:对于xBD数据集,使用在EuroSat卫星图像上预训练的CNN-LSTM模型与ResNet50骨干网络;对于RescueNet数据集,则使用了在UAV图片上预训练的Vision Transformer(ViT)模型。这两个系统都利用ConceptNet和WordNet等外部语义知识来扩展词汇覆盖率并提高描述准确性。 我们通过CLIPScore评估VLCE与其他领先的视觉-语言模型(LLaVA和QwenVL)之间的语义对齐,并使用InfoMetIC评估标题信息的丰富程度。实验结果表明,VLCE显著超越了基准模型,在InfoMetIC上达到了最高95.33%的同时保持了竞争性的语义对齐度。 我们的双架构系统在从卫星和无人机照片中自动生成具有行动力的信息密集型描述方面展示了巨大的潜力,从而提高灾害损害评估的效率。

URL

https://arxiv.org/abs/2509.21609

PDF

https://arxiv.org/pdf/2509.21609.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot