Abstract
Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking).
Abstract (translated)
目前最强的视频语言模型（VLMs）仍然是专有的。最强大的开源权重模型要么依赖于来自专有 VLM 的合成数据，从而进行蒸馏处理，要么不披露其训练数据或方法。因此，开源社区缺乏改进现有最佳视频（和图像）语言模型的基础。至关重要的是，许多下游应用不仅需要高层次的视频理解能力，还需要具备指针定位或像素跟踪的能力，而即使是专有模型也缺乏这种能力。 我们介绍了 Molmo2，这是一个新的 VLM 家族，在开源模型中处于领先位置，并在单张图片、多张图片和视频任务中的点驱动型定位方面展示了卓越的新功能。我们的关键贡献是一系列共 9 个新数据集（包括 7 个视频数据集及 2 个多图像数据集），其中包括用于预训练的详细描述性视频字幕的数据集，一个用于微调的自由形式视频问答数据集，一个新的具有复杂查询的对象跟踪数据集以及一项创新性的视频指针数据集，所有这些数据都是在不使用封闭 VLM 的情况下收集的。我们还提供了一种针对此数据集使用的训练方法，采用高效的打包和消息树编码方案，并展示出双向视觉标记注意力及新的标记权重策略可以提高性能。 我们的顶级 8B 模型在短视频、计数和描述等任务上优于同类开源权重和数据模型，在长视频上的表现也具有竞争力。在视频定位方面，Molmo2 显著超越现有的开源重量级模型（如 Qwen3-VL 的准确率为 35.5 对比 29.6 在视频计数中），并且在某些任务上甚至超过了专有模型（例如，在视频指针上的 F1 得分为 38.4 对比 Gemini 3 Pro 的 20.0，以及在视频跟踪上的 J&F 得分为 56.2 对比 Gemini 3 Pro 的 41.1）。
URL
https://arxiv.org/abs/2601.10611
