Paper Reading AI Learner

Team Xiaomi EV-AD VLA: Caption-Guided Retrieval System for Cross-Modal Drone Navigation - Technical Report for IROS 2025 RoboSense Challenge Track 4

2025-10-03 05:13:19
Lingfeng Zhang, Erjia Xiao, Yuchen Zhang, Haoxiang Fu, Ruibin Hu, Yanbiao Ma, Wenbo Ding, Long Chen, Hangjun Ye, Xiaoshuai Hao

Abstract

Cross-modal drone navigation remains a challenging task in robotics, requiring efficient retrieval of relevant images from large-scale databases based on natural language descriptions. The RoboSense 2025 Track 4 challenge addresses this challenge, focusing on robust, natural language-guided cross-view image retrieval across multiple platforms (drones, satellites, and ground cameras). Current baseline methods, while effective for initial retrieval, often struggle to achieve fine-grained semantic matching between text queries and visual content, especially in complex aerial scenes. To address this challenge, we propose a two-stage retrieval refinement method: Caption-Guided Retrieval System (CGRS) that enhances the baseline coarse ranking through intelligent reranking. Our method first leverages a baseline model to obtain an initial coarse ranking of the top 20 most relevant images for each query. We then use Vision-Language-Model (VLM) to generate detailed captions for these candidate images, capturing rich semantic descriptions of their visual content. These generated captions are then used in a multimodal similarity computation framework to perform fine-grained reranking of the original text query, effectively building a semantic bridge between the visual content and natural language descriptions. Our approach significantly improves upon the baseline, achieving a consistent 5\% improvement across all key metrics (Recall@1, Recall@5, and Recall@10). Our approach win TOP-2 in the challenge, demonstrating the practical value of our semantic refinement strategy in real-world robotic navigation scenarios.

Abstract (translated)

跨模态无人机导航在机器人领域仍是一项具有挑战性的任务,需要根据自然语言描述从大规模数据库中高效检索相关图像。RoboSense 2025 Track 4 挑战赛致力于解决这一问题,重点在于多平台(无人机、卫星和地面相机)上基于自然语言指导的跨视角图像检索。尽管现有的基准方法在初步检索方面效果良好,但它们往往难以实现文本查询与视觉内容之间细微语义匹配,尤其是在复杂的空中场景中。 为了解决这一挑战,我们提出了一种两阶段检索优化方法:Caption-Guided Retrieval System(CGRS)。该系统通过智能重排名改进了基准粗排的结果。具体来说,我们的方法首先利用一个基准模型获取每个查询的前20张最相关图像的初步粗略排序。接着,使用视觉-语言模型(VLM)为这些候选图像生成详细的描述性字幕,捕捉它们视觉内容中的丰富语义信息。随后,在多模态相似度计算框架下利用这些生成的字幕对原始文本查询进行精细重排名,有效地在视觉内容和自然语言描述之间建立了一座语义桥梁。 我们的方法显著超越了基准模型,在所有关键指标(Recall@1、Recall@5 和 Recall@10)上均实现了平均5%的性能提升。我们在挑战赛中获得了第二名的成绩,这充分展示了我们提出的语义优化策略在现实世界中的机器人导航场景中的实用价值。 通过上述方法和实验结果,可以发现基于生成性描述与多模态相似度计算的重排名机制能够有效增强图像检索模型的语义理解能力,在实际应用中具有重要的参考意义。

URL

https://arxiv.org/abs/2510.02728

PDF

https://arxiv.org/pdf/2510.02728.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot