Abstract
Multilingual encoder-based language models are widely adopted for code-mixed analysis tasks, yet we know surprisingly little about how they represent code-mixed inputs internally - or whether those representations meaningfully connect to the constituent languages being mixed. Using Hindi-English as a case study, we construct a unified trilingual corpus of parallel English, Hindi (Devanagari), and Romanized code-mixed sentences, and probe cross-lingual representation alignment across standard multilingual encoders and their code-mixed adapted variants via CKA, token-level saliency, and entropy-based uncertainty analysis. We find that while standard models align English and Hindi well, code-mixed inputs remain loosely connected to either language - and that continued pre-training on code-mixed data improves English-code-mixed alignment at the cost of English-Hindi alignment. Interpretability analyses further reveal a clear asymmetry: models process code-mixed text through an English-dominant semantic subspace, while native-script Hindi provides complementary signals that reduce representational uncertainty. Motivated by these findings, we introduce a trilingual post-training alignment objective that brings code-mixed representations closer to both constituent languages simultaneously, yielding more balanced cross-lingual alignment and downstream gains on sentiment analysis and hate speech detection - showing that grounding code-mixed representations in their constituent languages meaningfully helps cross-lingual understanding.
Abstract (translated)
基于多语言编码器的语言模型被广泛用于代码混合分析任务,然而我们对其内部如何表征代码混合输入,或这些表征是否与混合的组成语言存在有意义的关联,却知之甚少。本文以印地语-英语为案例,构建了一个包含英语、印地语(天城文)及罗马化代码混合句子的统一三语平行语料库,并通过中心核对齐(CKA)、词元级显著性分析及基于熵的不确定性分析,探测标准多语言编码器及其代码混合适配变体间的跨语言表征对齐情况。研究发现:尽管标准模型能良好对齐英语与印地语,但代码混合输入与任一语言的关联均较为松散;且在代码混合数据上持续预训练会以牺牲英语-印地语对齐为代价,提升英语-代码混合对齐。可解释性分析进一步揭示明显的不对称性:模型通过英语主导的语义子空间处理代码混合文本,而原生文字印地语则提供互补信号以降低表征不确定性。基于这些发现,我们提出一种三语后训练对齐目标,使代码混合表征同时贴近两种组成语言,从而实现更均衡的跨语言对齐,并在情感分析与仇恨言论检测任务中获得下游性能提升——表明将代码混合表征锚定于其组成语言,能有效促进跨语言理解。
URL
https://arxiv.org/abs/2603.19771