Abstract
Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model's parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X, a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.
Abstract (translated)
有效的跨语言迁移仍然是扩大大型语言模型从高资源语言到低资源语言效益的关键挑战。为了解决这个问题，先前的研究探索了多种方法来结合任务特定数据（在高资源源语言中）中的任务知识和未标记文本（在低资源目标语言中）中的语言知识。其中一种值得注意的方法是提出了可组合的稀疏微调(SFT)，用于跨语言迁移，这种方法学习任务特定和语言特定的稀疏掩码来选择预训练模型参数的一个子集进行进一步的微调。这些稀疏微调向量（SFTs）随后与预训练模型结合使用，以促进仅通过源语言的任务特定数据就实现零样本跨语言迁移到目标语言中的任务。用于SFTs的稀疏掩码是通过简单的基于幅度的剪枝来识别的。 在我们的工作中，我们引入了DeFT-X，这是一种新的可组合SFT方法，在进行幅度剪枝之前使用奇异值分解对预训练模型的权重矩阵进行去噪处理，从而产生更健壮的SFTs。我们在多样化的极度低资源语言（用于情感分类NusaX和自然语言推理AmericasNLI）上评估了DeFT-X，并证明它在性能上与SFT和其他显着的跨语言迁移基线相当或优于它们。
URL
https://arxiv.org/abs/2505.15090
