Abstract
In various natural language processing (NLP) tasks, fine-tuning Pre-trained Language Models (PLMs) often leads to the issue of spurious correlations, which negatively impacts performance, particularly when dealing with out-of-distribution data. To address this problem, we propose SALAD}(Structure Aware and LLM-driven Augmented Data), a novel approach designed to enhance model robustness and generalization by generating structure-aware and counterfactually augmented data for contrastive learning. Our method leverages a tagging-based approach to generate structure-aware positive samples and utilizes large language models (LLMs) to generate counterfactual negative samples with diverse sentence patterns. By applying contrastive learning, SALAD enables the model to focus on learning the structural relationships between key sentence components while minimizing reliance on spurious correlations. We validate our approach through experiments on three tasks: Sentiment Classification, Sexism Detection, and Natural Language Inference. The results demonstrate that SALAD not only improves model robustness and performance across different environments but also enhances generalization to out-of-distribution datasets and cross-domain scenarios.
Abstract (translated)
在各种自然语言处理(NLP)任务中,对预训练语言模型(PLMs)进行微调往往会带来虚假相关性的问题,这会负面影响性能,特别是在处理分布外数据时。为了解决这个问题,我们提出了SALAD(结构感知与大规模语言模型驱动的增强数据),这是一种通过生成结构感知且反事实增强的数据来进行对比学习的方法,旨在提高模型的鲁棒性和泛化能力。我们的方法采用基于标签的方法来生成结构感知的正样本,并利用大型语言模型(LLMs)生成具有多种句子模式的反事实负样本。通过应用对比学习,SALAD使模型能够专注于关键句法成分之间的结构性关系的学习,同时最小化对虚假相关性的依赖。 我们通过对三个任务——情感分类、性别歧视检测和自然语言推理进行实验来验证我们的方法的有效性。结果表明,SALAD不仅提高了在不同环境下的模型鲁棒性和性能,还增强了其对于分布外数据集以及跨域场景的泛化能力。
URL
https://arxiv.org/abs/2504.12185