Paper Reading AI Learner

TrackTeller: Temporal Multimodal 3D Grounding for Behavior-Dependent Object References

2025-12-25 12:02:56
Jiahong Yu, Ziqi Wang, Hailiang Zhao, Wei Zhai, Xueqiang Yan, Shuiguang Deng

Abstract

Understanding natural-language references to objects in dynamic 3D driving scenes is essential for interactive autonomous systems. In practice, many referring expressions describe targets through recent motion or short-term interactions, which cannot be resolved from static appearance or geometry alone. We study temporal language-based 3D grounding, where the objective is to identify the referred object in the current frame by leveraging multi-frame observations. We propose TrackTeller, a temporal multimodal grounding framework that integrates LiDAR-image fusion, language-conditioned decoding, and temporal reasoning in a unified architecture. TrackTeller constructs a shared UniScene representation aligned with textual semantics, generates language-aware 3D proposals, and refines grounding decisions using motion history and short-term dynamics. Experiments on the NuPrompt benchmark demonstrate that TrackTeller consistently improves language-grounded tracking performance, outperforming strong baselines with a 70% relative improvement in Average Multi-Object Tracking Accuracy and a 3.15-3.4 times reduction in False Alarm Frequency.

Abstract (translated)

理解动态三维驾驶场景中自然语言指向的对象对于交互式自主系统来说是至关重要的。在实际应用中,许多指称表达通过近期运动或短期互动来描述目标,这些信息无法仅凭静态外观或几何结构解析出来。本文研究了基于时间的语言驱动的3D定位问题,其目标是在利用多帧观察的基础上识别当前帧中的被指对象。 为此,我们提出了TrackTeller,这是一种整合LiDAR-图像融合、语言条件解码以及时间推理于一体的统一架构的时序多模态定位框架。TrackTeller构建了一个与文本语义对齐的共享UniScene表示,生成具有语言感知能力的3D提案,并利用运动历史和短期动态来优化定位决策。 在NuPrompt基准测试上的实验表明,TrackTeller能够显著提升基于语言的跟踪性能,相对于强大的基线方法,在平均多目标跟踪准确度方面有70%的相对改进,同时降低了误报频率(False Alarm Frequency),具体而言是减少了3.15到3.4倍。

URL

https://arxiv.org/abs/2512.21641

PDF

https://arxiv.org/pdf/2512.21641.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot