Abstract
Multimodal object detection leveraging RGB and Infrared (IR) images is pivotal for robust perception in all-weather scenarios. While recent adapter-based approaches efficiently transfer RGB-pretrained foundation models to this task, they often prioritize model efficiency at the expense of cross-modal structural consistency. Consequently, critical structural cues are frequently lost when significant domain gaps arise, such as in high-contrast or nighttime environments. Moreover, conventional static multimodal fusion mechanisms typically lack environmental awareness, resulting in suboptimal adaptation and constrained detection performance under complex, dynamic scene variations. To address these limitations, we propose SLGNet, a parameter-efficient framework that synergizes hierarchical structural priors and language-guided modulation within a frozen Vision Transformer (ViT)-based foundation model. Specifically, we design a Structure-Aware Adapter to extract hierarchical structural representations from both modalities and dynamically inject them into the ViT to compensate for structural degradation inherent in ViT-based backbones. Furthermore, we propose a Language-Guided Modulation module that exploits VLM-driven structured captions to dynamically recalibrate visual features, thereby endowing the model with robust environmental awareness. Extensive experiments on the LLVIP, FLIR, KAIST, and DroneVehicle datasets demonstrate that SLGNet establishes new state-of-the-art performance. Notably, on the LLVIP benchmark, our method achieves an mAP of 66.1, while reducing trainable parameters by approximately 87% compared to traditional full fine-tuning. This confirms SLGNet as a robust and efficient solution for multimodal perception.
Abstract (translated)
基于RGB和红外（IR）图像的多模态物体检测对于全天气条件下的稳健感知至关重要。尽管最近采用适配器方法可以高效地将预训练在RGB图像上的基础模型应用于此类任务，但这些方法通常以牺牲跨模式结构一致性为代价来提高模型效率。因此，在高对比度或夜间等显著领域差距环境中，关键的结构线索常常会丢失。此外，传统的静态多模态融合机制往往缺乏环境感知能力，导致在复杂、动态场景变化下适应性差和检测性能受限。 为了克服这些限制，我们提出了SLGNet框架，这是一个参数高效的模型，它结合了基于冻结视觉变换器（ViT）基础模型内的层级结构先验与语言引导调制。具体而言，我们设计了一种“感知结构适配器”，以提取两种模态的分层结构表示，并将其动态注入到ViT中，以补偿ViT主干固有的结构退化问题。此外，我们还提出了一种“语言引导调节”模块，利用视觉-语言模型驱动的结构化说明来动态调整视觉特征，从而赋予模型强大的环境感知能力。 在LLVIP、FLIR、KAIST和DroneVehicle数据集上的广泛实验表明，SLGNet建立了新的性能标杆。特别是在LLVIP基准测试中，我们的方法实现了66.1的mAP值，并且相比于传统的全微调训练参数，可减少约87%的可训练参数数量。这证实了SLGNet是一个稳健而高效的多模态感知解决方案。 这种创新的方法不仅提高了模型在不同环境条件下的适应性和鲁棒性，而且通过更有效的参数利用实现了更高的检测精度和效率，为全天气条件下的物体识别提供了强有力的支持。
URL
https://arxiv.org/abs/2601.02249
