Paper Reading AI Learner

CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

2026-02-23 08:47:19
Chunlei Meng, Guanhong Huang, Rong Fu, Runmin Jian, Zhongxue Gan, Chun Ouyang

Abstract

Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.

Abstract (translated)

跨模态学习旨在捕捉来自多种模态的共享和私有信息。然而,现有的将所有模态投影到单一潜在空间中进行融合的方法往往忽略了多模态数据中的异步、多层次语义结构。这种忽视导致了语义对齐问题和错误传播,从而降低了表示的质量。为了解决这个问题,我们提出了跨层次协同表征(CLCR),该方法明确地将每个模态的特征组织成三个级别的语义层级,并在每一层中规定跨模态交互的约束条件。 首先,语义层级编码器对各模态中的浅层、中间和深层特征进行对齐,建立共同的基础以促进互操作性。然后,在每一个层次上,通过引入跨层次协同交换域(IntraCED),将特征分解为共享子空间和私有子空间,并通过一个可学习的令牌预算来限制跨模态注意机制仅关注共享子空间。这种设计确保了只有共享语义被交换,并防止了从私有通道泄露信息。 为了在不同层次之间整合信息,我们引入了一个跨层次协同聚合域(InterCAD),它利用已学得的锚点同步各个级别的语义规模,在每个层级选择性地融合共享表征并用门控机制处理私有线索以形成一个简洁的任务表示。此外,我们还引入了正则化项来强制执行共享和私有特征之间的分离,并最小化跨层次干扰。 在六个涵盖情感识别、事件定位、情绪分析以及动作识别的基准测试上进行的实验表明,CLCR实现了强大的性能并在不同任务中表现出了良好的泛化能力。

URL

https://arxiv.org/abs/2602.19605

PDF

https://arxiv.org/pdf/2602.19605.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot