Abstract
Visual geo-localization for drones faces critical degradation under weather perturbations, \eg, rain and fog, where existing methods struggle with two inherent limitations: 1) Heavy reliance on limited weather categories that constrain generalization, and 2) Suboptimal disentanglement of entangled scene-weather features through pseudo weather categories. We present WeatherPrompt, a multi-modality learning paradigm that establishes weather-invariant representations through fusing the image embedding with the text context. Our framework introduces two key contributions: First, a Training-free Weather Reasoning mechanism that employs off-the-shelf large multi-modality models to synthesize multi-weather textual descriptions through human-like reasoning. It improves the scalability to unseen or complex weather, and could reflect different weather strength. Second, to better disentangle the scene and weather feature, we propose a multi-modality framework with the dynamic gating mechanism driven by the text embedding to adaptively reweight and fuse visual features across modalities. The framework is further optimized by the cross-modal objectives, including image-text contrastive learning and image-text matching, which maps the same scene with different weather conditions closer in the respresentation space. Extensive experiments validate that, under diverse weather conditions, our method achieves competitive recall rates compared to state-of-the-art drone geo-localization methods. Notably, it improves Recall@1 by +13.37\% under night conditions and by 18.69\% under fog and snow conditions.
Abstract (translated)
无人机视觉地理定位在天气变化(如雨和雾)条件下会面临严重的性能退化,现有方法存在两个根本性的局限:1) 过度依赖有限的天气类别,限制了其泛化能力;2) 对纠缠在一起的场景与天气特征解耦不够理想,通过伪天气类别来处理。我们提出了WeatherPrompt,这是一种多模态学习范式,通过融合图像嵌入和文本上下文来建立不受天气影响的表现形式。 我们的框架有两个关键贡献: 1. **无训练天气推理机制**:利用现成的大型多模态模型,通过类似人类的推理合成多种天气条件下的文字描述。这改善了对未见过或复杂天气情况的适应能力,并能反映不同强度的天气变化。 2. 为了更好地分离场景和天气特征,我们提出了一种基于文本嵌入驱动的动态门控机制多模态框架,该机制能够自适应地重新加权并融合跨模式的视觉特性。通过图像-文本对比学习和图像-文本匹配等跨模态目标进一步优化框架,使得在不同天气条件下相同的场景能够在表示空间中更接近。 广泛的实验验证了我们的方法在各种天气条件下均能达到与当前最先进的无人机地理定位方法相当或更高的召回率:特别是在夜间情况下提高了13.37%,而在雾和雪的情况下提升了18.69%。
URL
https://arxiv.org/abs/2508.09560