Abstract
Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.
Abstract (translated)
跨模态学习旨在捕捉来自多种模态的共享和私有信息。然而,现有的将所有模态投影到单一潜在空间中进行融合的方法往往忽略了多模态数据中的异步、多层次语义结构。这种忽视导致了语义对齐问题和错误传播,从而降低了表示的质量。为了解决这个问题,我们提出了跨层次协同表征(CLCR),该方法明确地将每个模态的特征组织成三个级别的语义层级,并在每一层中规定跨模态交互的约束条件。 首先,语义层级编码器对各模态中的浅层、中间和深层特征进行对齐,建立共同的基础以促进互操作性。然后,在每一个层次上,通过引入跨层次协同交换域(IntraCED),将特征分解为共享子空间和私有子空间,并通过一个可学习的令牌预算来限制跨模态注意机制仅关注共享子空间。这种设计确保了只有共享语义被交换,并防止了从私有通道泄露信息。 为了在不同层次之间整合信息,我们引入了一个跨层次协同聚合域(InterCAD),它利用已学得的锚点同步各个级别的语义规模,在每个层级选择性地融合共享表征并用门控机制处理私有线索以形成一个简洁的任务表示。此外,我们还引入了正则化项来强制执行共享和私有特征之间的分离,并最小化跨层次干扰。 在六个涵盖情感识别、事件定位、情绪分析以及动作识别的基准测试上进行的实验表明,CLCR实现了强大的性能并在不同任务中表现出了良好的泛化能力。
URL
https://arxiv.org/abs/2602.19605