Abstract
The computer vision domain has greatly benefited from an abundance of data across many modalities to improve on various visual tasks. Recently, there has been a lot of focus on self-supervised pre-training methods through Masked Autoencoders (MAE) \cite{he2022masked,bachmann2022multimae}, usually used as a first step before optimizing for a downstream task, such as classification or regression. This is very useful as it doesn't require any manually labeled data. In this work, we introduce Probabilistic Hyper-Graphs using Masked Autoencoders (PHG-MAE): a novel model that unifies the classical work on neural graphs \cite{leordeanu2021semi} with the modern approach of masked autoencoders under a common theoretical framework. Through random masking of entire modalities, not just patches, the model samples from the distribution of hyper-edges on each forward pass. Additionally, the model adapts the standard MAE algorithm by combining pre-training and fine-tuning into a single training loop. Moreover, our approach enables the creation of inference-time ensembles which, through aggregation, boost the final prediction performance and consistency. Lastly, we show that we can apply knowledge distillation on top of the ensembles with little loss in performance, even with models that have fewer than 1M parameters. While our work mostly focuses on outdoor UAV scenes that contain multiple world interpretations and modalities, the same steps can be followed in other similar domains, such as autonomous driving or indoor robotics. In order to streamline the process of integrating external pre-trained experts for computer vision multi-modal multi-task learning (MTL) scenarios, we developed a data-pipeline software. Using this tool, we have created and released a fully-automated extension of the Dronescapes dataset. All the technical details, code and reproduction steps are publicly released.
Abstract (translated)
计算机视觉领域从各种模态中获取的大量数据中受益匪浅,这有助于提高各类视觉任务的表现。最近,人们越来越多地关注通过掩码自编码器(MAE)进行自我监督预训练方法的应用,这些方法通常在针对分类或回归等下游任务优化之前使用。这种方法非常有用,因为它不需要任何人工标注的数据。在此项工作中,我们引入了基于掩码自编码器的概率超图模型(PHG-MAE),这是一种新颖的模型,它将经典的神经图研究工作与现代掩码自编码器方法在统一的理论框架下进行了融合。通过随机遮蔽整个模态而非只是图像块,在每次前向传播过程中,该模型从超边分布中采样数据。此外,该模型还对标准MAE算法进行了改进,将预训练和微调结合到了一个单一的训练循环中。我们的方法还能在推理阶段创建集成模型,通过聚合操作提高最终预测性能的一致性。最后,我们展示了可以在这些集合上应用知识蒸馏技术,并且即使对于参数数量少于1M的小型模型而言,性能也不会有显著损失。虽然我们的工作主要集中在含有多种世界解释和模态的室外无人机场景上,但类似的步骤也可以应用于其他类似领域,比如自动驾驶或室内机器人学。 为了简化计算机视觉多模态多任务学习(MTL)场景中集成外部预训练专家的过程,我们开发了一款数据管道软件。利用这一工具,我们创建并发布了Dronescapes数据集的全自动化扩展版本。所有技术细节、代码和复现步骤均已公开发布。
URL
https://arxiv.org/abs/2510.10068