Abstract
Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.
Abstract (translated)
视觉语言模型(VLMs)在计数任务中表现出持续的幻觉,其准确性远低于其他视觉推理任务(不包括情感分析)。即使是在最先进的具备推理能力的VLM中,这一现象仍然存在。相比之下,基于CNN的对象检测模型(ODMs),如YOLO,在空间定位和实例计数方面表现出色,并且计算开销极小。我们提出了一种名为GroundCount的框架,该框架通过结合来自ODM的显式空间接地信息来减轻VLM中的计数幻觉问题。 在最佳情况下,我们的基于提示的增强策略使表现最好的模型(Ovis2.5-2B)的计数准确率达到了81.3%,相较于原模型提高了6.6个百分点,并通过消除导致推理循环的幻觉驱动推理,将更强模型的推断时间减少了22%。我们进行了全面的消融研究,表明位置编码是关键组件:对于较强的模型是有益的,但对于较弱的模型则有害。另一方面,置信度分数在大多数架构中引入了噪音,并且移除它们改善了其中四个评估模型的表现。 我们还评估了特征级融合架构,发现通过结构化提示实现的显式符号接地优于复杂的交叉注意力机制所驱动的隐性特征融合。我们的方法在五个评估的VLM架构中的四个(6.2--7.5个百分点)上带来了持续改进,而一个由于其迭代反射机制与结构化提示不兼容而导致性能下降。 这些结果表明,计数失败源自于基本的空间语义集成限制,而非特定体系结构缺陷,并强调了增强策略中体系结构兼容性的重要性。
URL
https://arxiv.org/abs/2603.10978