Paper Reading AI Learner

GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations

2026-03-11 17:04:30
Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique

Abstract

Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.

Abstract (translated)

视觉语言模型(VLMs)在计数任务中表现出持续的幻觉,其准确性远低于其他视觉推理任务(不包括情感分析)。即使是在最先进的具备推理能力的VLM中,这一现象仍然存在。相比之下,基于CNN的对象检测模型(ODMs),如YOLO,在空间定位和实例计数方面表现出色,并且计算开销极小。我们提出了一种名为GroundCount的框架,该框架通过结合来自ODM的显式空间接地信息来减轻VLM中的计数幻觉问题。 在最佳情况下,我们的基于提示的增强策略使表现最好的模型(Ovis2.5-2B)的计数准确率达到了81.3%,相较于原模型提高了6.6个百分点,并通过消除导致推理循环的幻觉驱动推理,将更强模型的推断时间减少了22%。我们进行了全面的消融研究,表明位置编码是关键组件:对于较强的模型是有益的,但对于较弱的模型则有害。另一方面,置信度分数在大多数架构中引入了噪音,并且移除它们改善了其中四个评估模型的表现。 我们还评估了特征级融合架构,发现通过结构化提示实现的显式符号接地优于复杂的交叉注意力机制所驱动的隐性特征融合。我们的方法在五个评估的VLM架构中的四个(6.2--7.5个百分点)上带来了持续改进,而一个由于其迭代反射机制与结构化提示不兼容而导致性能下降。 这些结果表明,计数失败源自于基本的空间语义集成限制,而非特定体系结构缺陷,并强调了增强策略中体系结构兼容性的重要性。

URL

https://arxiv.org/abs/2603.10978

PDF

https://arxiv.org/pdf/2603.10978.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot