Abstract
Despite notable advancements in remote sensing vision-language models (VLMs), existing models often struggle with spatial understanding, limiting their effectiveness in real-world applications. To push the boundaries of VLMs in remote sensing, we specifically address vehicle imagery captured by drones and introduce a spatially-aware dataset AirSpatial, which comprises over 206K instructions and introduces two novel tasks: Spatial Grounding and Spatial Question Answering. It is also the first remote sensing grounding dataset to provide 3DBB. To effectively leverage existing image understanding of VLMs to spatial domains, we adopt a two-stage training strategy comprising Image Understanding Pre-training and Spatial Understanding Fine-tuning. Utilizing this trained spatially-aware VLM, we develop an aerial agent, AirSpatialBot, which is capable of fine-grained vehicle attribute recognition and retrieval. By dynamically integrating task planning, image understanding, spatial understanding, and task execution capabilities, AirSpatialBot adapts to diverse query requirements. Experimental results validate the effectiveness of our approach, revealing the spatial limitations of existing VLMs while providing valuable insights. The model, code, and datasets will be released at this https URL
Abstract (translated)
尽管远程传感视觉-语言模型（VLM）取得了显著进展，但现有模型在空间理解方面仍面临挑战，限制了其在实际应用中的有效性。为了推动远程传感领域中VLM的边界，我们特别关注由无人机捕捉到的车辆图像，并引入了一个具有空间意识的数据集AirSpatial，该数据集中包含超过206K条指令，并提出了两个新颖的任务：空间定位（Spatial Grounding）和空间问答（Spatial Question Answering）。这也是首个提供3DBB信息的远程传感定位数据集。为了有效利用现有VLM在图像理解方面的知识并将其应用到空间领域，我们采用了一种两阶段训练策略，包括图像理解预训练和空间理解微调。借助这一经过训练的空间感知VLM，我们开发了一个空中代理AirSpatialBot，它可以进行细粒度的车辆属性识别与检索。通过动态整合任务规划、图像理解、空间理解和任务执行能力，AirSpatialBot能够适应各种查询需求。实验结果验证了该方法的有效性，揭示了现有VLM在空间理解方面的局限，并提供了有价值的见解。模型、代码和数据集将在以下网址发布：[this URL]
URL
https://arxiv.org/abs/2601.01416
