Abstract
This study proposes a lightweight multimodal adaptation framework to bridge the representation gap between RGB-pretrained VLMs and thermal infrared imagery, and demonstrates its practical utility using a real drone-collected dataset. A thermal dataset was developed from drone-collected imagery and was used to fine-tune VLMs through multimodal projector alignment, enabling the transfer of information from RGB-based visual representations to thermal radiometric inputs. Three representative models, including InternVL3-8B-Instruct, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct, were benchmarked under both closed-set and open-set prompting conditions for species recognition and instance enumeration. Among the tested models, Qwen3-VL-8B-Instruct with open-set prompting achieved the best overall performance, with F1 scores of 0.935 for deer, 0.915 for rhino, and 0.968 for elephant, and within-1 enumeration accuracies of 0.779, 0.982, and 1.000, respectively. In addition, combining thermal imagery with simultaneously collected RGB imagery enabled the model to generate habitat-context information, including land-cover characteristics, key landscape features, and visible human disturbance. Overall, the findings demonstrate that lightweight projector-based adaptation provides an effective and practical route for transferring RGB-pretrained VLMs to thermal drone imagery, expanding their utility from object-level recognition to habitat-context interpretation in ecological monitoring.
Abstract (translated)
本研究提出了一种轻量级多模态适应框架，用于弥合基于RGB预训练视觉语言模型与热红外影像之间的表征差距，并利用真实无人机采集数据集验证了其实用价值。研究从无人机影像中构建了热红外数据集，并通过多模态投影器对齐对视觉语言模型进行微调，实现了从基于RGB的视觉表征到热辐射测量输入的迁移。在封闭集与开放集提示条件下，对InternVL3-8B-Instruct、Qwen2.5-VL-7B-Instruct和Qwen3-VL-8B-Instruct三个代表性模型进行了物种识别与个体计数的基准测试。其中，采用开放集提示的Qwen3-VL-8B-Instruct模型综合表现最佳，在鹿、犀牛和大象的识别中F1分数分别达到0.935、0.915和0.968，误差在1以内的计数准确率分别为0.779、0.982和1.000。此外，结合同步采集的热红外与RGB影像，模型能够生成包含土地覆盖特征、关键景观要素及可见人为干扰的生境背景信息。总体而言，研究结果表明，基于投影器的轻量级适应方法为将RGB预训练视觉语言模型迁移至热红外无人机影像提供了有效且实用的路径，使其应用范围从目标级识别拓展至生态监测中的生境背景解读。
URL
https://arxiv.org/abs/2604.06124
