Paper Reading AI Learner

Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning

2025-10-11 07:05:34
P\^irvu Mihai-Cristian, Leordeanu Marius

Abstract

The computer vision domain has greatly benefited from an abundance of data across many modalities to improve on various visual tasks. Recently, there has been a lot of focus on self-supervised pre-training methods through Masked Autoencoders (MAE) \cite{he2022masked,bachmann2022multimae}, usually used as a first step before optimizing for a downstream task, such as classification or regression. This is very useful as it doesn't require any manually labeled data. In this work, we introduce Probabilistic Hyper-Graphs using Masked Autoencoders (PHG-MAE): a novel model that unifies the classical work on neural graphs \cite{leordeanu2021semi} with the modern approach of masked autoencoders under a common theoretical framework. Through random masking of entire modalities, not just patches, the model samples from the distribution of hyper-edges on each forward pass. Additionally, the model adapts the standard MAE algorithm by combining pre-training and fine-tuning into a single training loop. Moreover, our approach enables the creation of inference-time ensembles which, through aggregation, boost the final prediction performance and consistency. Lastly, we show that we can apply knowledge distillation on top of the ensembles with little loss in performance, even with models that have fewer than 1M parameters. While our work mostly focuses on outdoor UAV scenes that contain multiple world interpretations and modalities, the same steps can be followed in other similar domains, such as autonomous driving or indoor robotics. In order to streamline the process of integrating external pre-trained experts for computer vision multi-modal multi-task learning (MTL) scenarios, we developed a data-pipeline software. Using this tool, we have created and released a fully-automated extension of the Dronescapes dataset. All the technical details, code and reproduction steps are publicly released.

Abstract (translated)

计算机视觉领域从各种模态中获取的大量数据中受益匪浅,这有助于提高各类视觉任务的表现。最近,人们越来越多地关注通过掩码自编码器(MAE)进行自我监督预训练方法的应用,这些方法通常在针对分类或回归等下游任务优化之前使用。这种方法非常有用,因为它不需要任何人工标注的数据。在此项工作中,我们引入了基于掩码自编码器的概率超图模型(PHG-MAE),这是一种新颖的模型,它将经典的神经图研究工作与现代掩码自编码器方法在统一的理论框架下进行了融合。通过随机遮蔽整个模态而非只是图像块,在每次前向传播过程中,该模型从超边分布中采样数据。此外,该模型还对标准MAE算法进行了改进,将预训练和微调结合到了一个单一的训练循环中。我们的方法还能在推理阶段创建集成模型,通过聚合操作提高最终预测性能的一致性。最后,我们展示了可以在这些集合上应用知识蒸馏技术,并且即使对于参数数量少于1M的小型模型而言,性能也不会有显著损失。虽然我们的工作主要集中在含有多种世界解释和模态的室外无人机场景上,但类似的步骤也可以应用于其他类似领域,比如自动驾驶或室内机器人学。 为了简化计算机视觉多模态多任务学习(MTL)场景中集成外部预训练专家的过程,我们开发了一款数据管道软件。利用这一工具,我们创建并发布了Dronescapes数据集的全自动化扩展版本。所有技术细节、代码和复现步骤均已公开发布。

URL

https://arxiv.org/abs/2510.10068

PDF

https://arxiv.org/pdf/2510.10068.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot