Paper Reading AI Learner

WeatherPrompt: Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localization

2025-08-13 07:28:41
Jiahao Wen, Hang Yu, Zhedong Zheng

Abstract

Visual geo-localization for drones faces critical degradation under weather perturbations, \eg, rain and fog, where existing methods struggle with two inherent limitations: 1) Heavy reliance on limited weather categories that constrain generalization, and 2) Suboptimal disentanglement of entangled scene-weather features through pseudo weather categories. We present WeatherPrompt, a multi-modality learning paradigm that establishes weather-invariant representations through fusing the image embedding with the text context. Our framework introduces two key contributions: First, a Training-free Weather Reasoning mechanism that employs off-the-shelf large multi-modality models to synthesize multi-weather textual descriptions through human-like reasoning. It improves the scalability to unseen or complex weather, and could reflect different weather strength. Second, to better disentangle the scene and weather feature, we propose a multi-modality framework with the dynamic gating mechanism driven by the text embedding to adaptively reweight and fuse visual features across modalities. The framework is further optimized by the cross-modal objectives, including image-text contrastive learning and image-text matching, which maps the same scene with different weather conditions closer in the respresentation space. Extensive experiments validate that, under diverse weather conditions, our method achieves competitive recall rates compared to state-of-the-art drone geo-localization methods. Notably, it improves Recall@1 by +13.37\% under night conditions and by 18.69\% under fog and snow conditions.

Abstract (translated)

无人机视觉地理定位在天气变化(如雨和雾)条件下会面临严重的性能退化,现有方法存在两个根本性的局限:1) 过度依赖有限的天气类别,限制了其泛化能力;2) 对纠缠在一起的场景与天气特征解耦不够理想,通过伪天气类别来处理。我们提出了WeatherPrompt,这是一种多模态学习范式,通过融合图像嵌入和文本上下文来建立不受天气影响的表现形式。 我们的框架有两个关键贡献: 1. **无训练天气推理机制**:利用现成的大型多模态模型,通过类似人类的推理合成多种天气条件下的文字描述。这改善了对未见过或复杂天气情况的适应能力,并能反映不同强度的天气变化。 2. 为了更好地分离场景和天气特征,我们提出了一种基于文本嵌入驱动的动态门控机制多模态框架,该机制能够自适应地重新加权并融合跨模式的视觉特性。通过图像-文本对比学习和图像-文本匹配等跨模态目标进一步优化框架,使得在不同天气条件下相同的场景能够在表示空间中更接近。 广泛的实验验证了我们的方法在各种天气条件下均能达到与当前最先进的无人机地理定位方法相当或更高的召回率:特别是在夜间情况下提高了13.37%,而在雾和雪的情况下提升了18.69%。

URL

https://arxiv.org/abs/2508.09560

PDF

https://arxiv.org/pdf/2508.09560.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot