The growing adoption of machine learning (ML) in sensitive domains has heightened the demand for transparent and interpretable artificial intelligence. Large Language Models (LLMs) are increasingly capable of producing natural language explanations, yet it remains unclear whether these rationales faithfully capture the predictive signals that underlie decisions. This paper introduces RACE-Reasoning Alignment for Completeness of Explanations, a systematic framework to evaluate the alignment between LLM-generated explanations and interpretable feature importance scores derived from a logistic regression baseline. We analyze four widely used text classification datasets-WIKI ONTOLOGY, AG NEWS, IMDB, and GOEMOTIONS-and compare LLM rationales against top-ranked supporting and contradicting lexical features. To capture alignment at multiple levels of granularity, RACE implements token-aware, exact string, and edit-distance matching techniques. Empirical results reveal a consistent asymmetry: correct predictions exhibit higher coverage of supporting features, while incorrect predictions are associated with elevated coverage of contradicting features. Edit-distance matching further uncovers paraphrastic overlaps, boosting coverage while preserving this asymmetry. These findings demonstrate that LLM rationales combine both surface-level and flexible evidence reuse, yet can also amplify misleading cues in error cases. RACE provides new insights into the faithfulness of LLM explanations and establishes a quantitative basis for evaluating reasoning completeness in neural language models.
https://arxiv.org/abs/2510.21884
Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles, an efficient transformer-based ensemble that prunes attention heads to create diverse members and merges them via a new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Experiments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, in zero-shot classification on ImageNet-1k, our approach surpasses state of the art methods, even without requiring additional training.
不确定性量化(UQ)对于在安全关键环境中部署深度神经网络至关重要。尽管像Deep Ensembles这样的方法能够实现强大的不确定性量化性能,但它们的高计算和内存成本阻碍了其在大型模型中的可扩展性。我们引入了一种高效的基于Transformer的集合——Hydra Ensembles,它通过修剪注意力头来创建多样化的成员,并利用一种新的具有分组全连接层的多头注意力机制将这些成员合并起来。这种方法生成了一个紧凑的模型,其推理速度接近单个网络的速度,在不从头开始重新训练的情况下,不确定性量化的性能可以与Deep Ensembles相匹配甚至超越。 我们还提供了对修剪方法的深入分析,表明简单的修剪方式可能会损害校准效果,而Hydra Ensembles则能够保持强大的不确定性估计。在图像和文本分类任务中进行的各种架构实验显示,与Deep Ensembles相比,我们的方法具有持续的优势。尤其值得注意的是,在零样本(zero-shot)ImageNet-1k分类测试中,即使不增加额外的训练数据,我们提出的方法也超越了现有的最佳方法。
https://arxiv.org/abs/2510.18358
The exponential increase in scientific literature and online information necessitates efficient methods for extracting knowledge from textual data. Natural language processing (NLP) plays a crucial role in addressing this challenge, particularly in text classification tasks. While large language models (LLMs) have achieved remarkable success in NLP, their accuracy can suffer in domain-specific contexts due to specialized vocabulary, unique grammatical structures, and imbalanced data distributions. In this systematic literature review (SLR), we investigate the utilization of pre-trained language models (PLMs) for domain-specific text classification. We systematically review 41 articles published between 2018 and January 2024, adhering to the PRISMA statement (preferred reporting items for systematic reviews and meta-analyses). This review methodology involved rigorous inclusion criteria and a multi-step selection process employing AI-powered tools. We delve into the evolution of text classification techniques and differentiate between traditional and modern approaches. We emphasize transformer-based models and explore the challenges and considerations associated with using LLMs for domain-specific text classification. Furthermore, we categorize existing research based on various PLMs and propose a taxonomy of techniques used in the field. To validate our findings, we conducted a comparative experiment involving BERT, SciBERT, and BioBERT in biomedical sentence classification. Finally, we present a comparative study on the performance of LLMs in text classification tasks across different domains. In addition, we examine recent advancements in PLMs for domain-specific text classification and offer insights into future directions and limitations in this rapidly evolving domain.
科学文献和在线信息的指数增长要求高效的方法从文本数据中提取知识。自然语言处理(NLP)在应对这一挑战方面发挥着关键作用,特别是在文本分类任务中。尽管大型语言模型(LLMs)在NLP领域取得了显著的成功,但在特定领域的背景下,由于专业词汇、独特的语法结构以及不平衡的数据分布等原因,它们的准确性可能会受到影响。在这次系统性文献综述(SLR)中,我们研究了预训练语言模型(PLMs)在特定领域文本分类中的应用。我们在2018年至2024年1月间共审查了41篇论文,并遵循PRISMA声明(系统评价和元分析的首选报告项目)。该综述方法采用了严格的纳入标准以及使用AI工具进行多步骤选择的过程。我们深入探讨了文本分类技术的发展历程,区分了传统方法与现代方法的区别。我们强调基于转换器的模型,并研究了在特定领域内应用LLMs时面临的挑战和考虑因素。此外,我们依据不同的PLMs对现有研究进行了分类,并提出了一种该领域的技术分类体系。为了验证我们的发现,我们在生物医学句子分类任务中进行了BERT、SciBERT和BioBERT之间的比较实验。最后,我们提供了不同领域中大型语言模型在文本分类任务性能上的对比分析。此外,我们还考察了PLMs在特定领域文本分类中的最新进展,并为该快速发展的领域的未来方向及限制性因素提供见解。
https://arxiv.org/abs/2510.17892
The surge in user-generated reviews has amplified the need for interpretable models that can provide fine-grained insights. Existing prototype-based models offer intuitive explanations but typically operate at coarse granularity (sentence or document level) and fail to address the multi-label nature of real-world text classification. We propose ProtoSiTex, a semi-interpretable framework designed for fine-grained multi-label text classification. ProtoSiTex employs a dual-phase alternating training strategy: an unsupervised prototype discovery phase that learns semantically coherent and diverse prototypes, and a supervised classification phase that maps these prototypes to class labels. A hierarchical loss function enforces consistency across sub-sentence, sentence, and document levels, enhancing interpretability and alignment. Unlike prior approaches, ProtoSiTex captures overlapping and conflicting semantics using adaptive prototypes and multi-head attention. We also introduce a benchmark dataset of hotel reviews annotated at the sub-sentence level with multiple labels. Experiments on this dataset and two public benchmarks (binary and multi-class) show that ProtoSiTex achieves state-of-the-art performance while delivering faithful, human-aligned explanations, establishing it as a robust solution for semi-interpretable multi-label text classification.
用户生成的评论数量激增,增强了对能够提供细致洞察的可解释模型的需求。现有的基于原型的模型提供了直观的解释,但通常以粗粒度(句子或文档级别)运行,并且无法解决现实世界文本分类中的多标签特性问题。我们提出了ProtoSiTex,这是一种面向细粒度多标签文本分类设计的半可解释框架。ProtoSiTex采用了一种双阶段交替训练策略:一个无监督原型发现阶段,该阶段学习语义上连贯且多样化的原型;以及一个有监督分类阶段,将这些原型映射到类别标签。层次化损失函数确保了在子句、句子和文档级别之间的一致性,从而增强了可解释性和对齐度。与先前的方法不同,ProtoSiTex使用自适应原型和多头注意力机制来捕捉重叠和冲突的语义。 我们还引入了一个基准数据集,该数据集包含酒店评论,并且这些评论在子句级别的多个标签上进行了标注。实验结果表明,在该数据集以及两个公开基准(二元分类和多类分类)上的测试中,ProtoSiTex实现了最先进的性能并提供了忠实、与人类一致的解释,从而确立了其作为半可解释多标签文本分类领域中的稳健解决方案的地位。
https://arxiv.org/abs/2510.12534
Text preprocessing is a fundamental component of Natural Language Processing, involving techniques such as stopword removal, stemming, and lemmatization to prepare text as input for further processing and analysis. Despite the context-dependent nature of the above techniques, traditional methods usually ignore contextual information. In this paper, we investigate the idea of using Large Language Models (LLMs) to perform various preprocessing tasks, due to their ability to take context into account without requiring extensive language-specific annotated resources. Through a comprehensive evaluation on web-sourced data, we compare LLM-based preprocessing (specifically stopword removal, lemmatization and stemming) to traditional algorithms across multiple text classification tasks in six European languages. Our analysis indicates that LLMs are capable of replicating traditional stopword removal, lemmatization, and stemming methods with accuracies reaching 97%, 82%, and 74%, respectively. Additionally, we show that ML algorithms trained on texts preprocessed by LLMs achieve an improvement of up to 6% with respect to the $F_1$ measure compared to traditional techniques. Our code, prompts, and results are publicly available at this https URL.
文本预处理是自然语言处理中的一个基本组成部分,涉及诸如停用词去除、词干提取和词形还原等技术,以准备作为进一步处理和分析输入的文本。尽管上述技术具有上下文依赖性,传统方法通常会忽略上下文信息。在本文中,我们探讨了使用大型语言模型(LLM)执行各种预处理任务的想法,因为它们能够在不需大量特定语言注释资源的情况下考虑到上下文。通过在六种欧洲语言的多个文本分类任务上的网络数据进行全面评估,我们将基于LLM的预处理(特别是停用词去除、词形还原和词干提取)与传统算法进行了比较。我们的分析表明,LLMs能够以高达97%、82%和74%的准确率复制传统的停用词去除、词形还原和词干提取方法。此外,我们还展示了在使用LLM预处理文本上进行训练的机器学习算法,在$F_1$度量方面相比传统技术提高了最多6%。 我们的代码、提示以及结果可在以下网址公开获取:[此链接](https://example.com)(请注意实际发表时请替换为正确的URL)。
https://arxiv.org/abs/2510.11482
We present a dataset of 833k paragraphs extracted from CC-BY licensed scientific publications, classified into four categories: acknowledgments, data mentions, software/code mentions, and clinical trial mentions. The paragraphs are primarily in English and French, with additional European languages represented. Each paragraph is annotated with language identification (using fastText) and scientific domain (from OpenAlex). This dataset, derived from the French Open Science Monitor corpus and processed using GROBID, enables training of text classification models and development of named entity recognition systems for scientific literature mining. The dataset is publicly available on HuggingFace this https URL under a CC-BY license.
https://arxiv.org/abs/2510.21762
Distance-based unsupervised text classification is a method within text classification that leverages the semantic similarity between a label and a text to determine label relevance. This method provides numerous benefits, including fast inference and adaptability to expanding label sets, as opposed to zero-shot, few-shot, and fine-tuned neural networks that require re-training in such cases. In multi-label distance-based classification and information retrieval algorithms, thresholds are required to determine whether a text instance is "similar" to a label or query. Similarity between a text and label is determined in a dense embedding space, usually generated by state-of-the-art sentence encoders. Multi-label classification complicates matters, as a text instance can have multiple true labels, unlike in multi-class or binary classification, where each instance is assigned only one label. We expand upon previous literature on this underexplored topic by thoroughly examining and evaluating the ability of sentence encoders to perform distance-based classification. First, we perform an exploratory study to verify whether the semantic relationships between texts and labels vary across models, datasets, and label sets by conducting experiments on a diverse collection of realistic multi-label text classification (MLTC) datasets. We find that similarity distributions show statistically significant differences across models, datasets and even label sets. We propose a novel method for optimizing label-specific thresholds using a validation set. Our label-specific thresholding method achieves an average improvement of 46% over normalized 0.5 thresholding and outperforms uniform thresholding approaches from previous work by an average of 14%. Additionally, the method demonstrates strong performance even with limited labeled examples.
基于距离的无监督文本分类是文本分类领域的一种方法,该方法利用标签与文本之间的语义相似性来判断标签的相关性。相比于零样本、小样本和微调神经网络需要重新训练的情况,这种方法提供了许多优势,包括快速推理以及对不断扩大的标签集具有良好的适应能力。 在多标签距离基于的分类及信息检索算法中,为了确定某个文本实例是否与一个标签或查询“相似”,都需要设定阈值。文本和标签之间的相似性是在由先进句子编码器生成的稠密嵌入空间内进行判定的。与多类或者二元分类不同,在多标签分类中,一个文本实例可能有多个真实标签。 我们在之前较少探索的主题上进行了拓展研究,详细地考察并评估了句子编码器在基于距离的分类中的性能。首先,我们通过一系列实验来验证模型、数据集和标签集合之间的语义关系是否存在差异。这些实验是在一组多样化的现实多标签文本分类(MLTC)数据集上进行的。我们发现相似性的分布统计上显著地不同,这表明了在不同的模型和数据集中存在着变化。 基于这一观察,我们提出了一个新的方法来利用验证集优化特定于标签的阈值。通过这种方法,相较于标准化后的0.5阈值法,我们的特定标签阈值设定平均提升了46%,并且相比于以往研究中的统一阈值设定策略,该方法也表现出平均14%的优势。此外,此方法即使在标注样本有限的情况下也能展现出强大的性能。 这种方法不仅提供了改进的分类准确性,还为复杂的多标签场景提供了解决方案,并且证明了针对不同数据集和标签优化特定阈值的重要性。
https://arxiv.org/abs/2510.11160
Parameter-efficient fine-tuning (PEFT) large language models (LLMs) have shown impressive performance in various downstream tasks. However, in many real-world scenarios, the collected training data inevitably contains noisy labels. To learn from noisy labels, most solutions select samples with small losses for model training. However, the selected samples, in turn, impact the loss computation in the next iteration. An inaccurate initial selection can create a vicious cycle, leading to suboptimal performance. To break this cycle, we propose Delora, a novel framework that decouples the sample selection from model training. For sample selection, Delora establishes a noisy label detector by introducing clean and noisy LoRA. Benefiting from the memory effect, the clean LoRA is encouraged to memorize clean data, while the noisy LoRA is constrained to memorize mislabeled data, which serves as a learnable threshold for selecting clean and noisy samples. For model training, Delora can use carefully selected samples to fine-tune language models seamlessly. Experimental results on synthetic and real-world noisy datasets demonstrate the effectiveness of Delora in noisy label detection and text classification.
参数高效的微调(PEFT)大型语言模型(LLMs)在各种下游任务中表现出令人印象深刻的性能。然而,在许多现实场景中,收集的训练数据不可避免地包含噪声标签。为从噪声标签中学习,大多数解决方案选择具有小损失值的数据样本进行模型训练。然而,所选样本反过来会影响下一次迭代中的损失计算。初始选择不准确可能会产生恶性循环,导致次优性能。为了打破这个循环,我们提出了 Delora,这是一个将样本选择与模型训练解耦的新型框架。 对于样本选择,Delora 通过引入干净和噪声LoRA建立了噪声标签检测器。得益于记忆效应,干净的 LoRA 被鼓励记住清洁数据,而噪声的 LoRA 则被限制记住错误标记的数据,这作为可学习的阈值用于区分清洁与噪声样本的选择。 对于模型训练,Delora 可以使用精心挑选的样本无缝地微调语言模型。在合成和真实世界的噪声数据集上的实验结果证明了 Delora 在噪声标签检测和文本分类方面的有效性。
https://arxiv.org/abs/2510.10208
Sentence embedding methods have made remarkable progress, yet they still struggle to capture the implicit semantics within sentences. This can be attributed to the inherent limitations of conventional sentence embedding methods that assign only a single vector per sentence. To overcome this limitation, we propose DualCSE, a sentence embedding method that assigns two embeddings to each sentence: one representing the explicit semantics and the other representing the implicit semantics. These embeddings coexist in the shared space, enabling the selection of the desired semantics for specific purposes such as information retrieval and text classification. Experimental results demonstrate that DualCSE can effectively encode both explicit and implicit meanings and improve the performance of the downstream task.
句子嵌入方法已经取得了显著的进步,但仍难以捕捉句子中的隐含语义。这可以归因于传统句子嵌入方法固有的限制,即每句话仅分配一个向量。为了克服这一局限性,我们提出了一种名为DualCSE的句子嵌入方法,该方法为每个句子分配两个嵌入:一个是表示显式语义的嵌入,另一个是表示隐含语义的嵌入。这些嵌入共存于共享空间中,从而可以在诸如信息检索和文本分类等特定用途下选择所需的语义。实验结果表明,DualCSE能够有效地编码显性和隐性含义,并提高下游任务的表现。
https://arxiv.org/abs/2510.09293
Prompt optimization aims to systematically refine prompts to enhance a language model's performance on specific tasks. Fairness detection in Terms of Service (ToS) clauses is a challenging legal NLP task that demands carefully crafted prompts to ensure reliable results. However, existing prompt optimization methods are often computationally expensive due to inefficient search strategies and costly prompt candidate scoring. In this paper, we propose a framework that combines Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to more effectively explore the prompt space while reducing evaluation costs. Experiments demonstrate that our approach achieves higher classification accuracy and efficiency than baseline methods under a constrained computation budget.
提示优化的目标是系统地改进提示,以增强语言模型在特定任务上的表现。服务条款(ToS)中的公平性检测是一项具有挑战性的法律自然语言处理(NLP)任务,需要精心设计的提示来确保可靠的结果。然而,现有的提示优化方法通常由于搜索策略低效和候选提示评分成本高而计算开销大。在这篇论文中,我们提出了一种结合蒙特卡洛树搜索(MCTS)与代理提示评估器的框架,旨在更有效地探索提示空间同时减少评价成本。实验表明,在给定的计算预算下,我们的方法比基线方法实现了更高的分类准确率和效率。
https://arxiv.org/abs/2510.08524
Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8% for NER and +8.8% for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.
在移动平台上部署自然语言处理(NLP)模型需要能够在多种应用场景中适应并保持内存和计算效率的模型。我们研究了预微调策略,以增强轻量级BERT类似编码器对两种基本NLP任务类别:命名实体识别(NER)和文本分类的适用性。虽然单独的任务预微调可以改善每个任务类别的下游性能,但我们发现单纯的多任务预微调会引入相互冲突的优化信号,从而降低整体表现。 为了解决这个问题,我们提出了一种基于主要任务LoRA模块的简单而有效的多任务预微调框架,这使得单一共享编码器主干能够通过模块化适配器实现。我们的方法在满足实际部署约束的同时,实现了与单独预微调相当的表现水平。在21个下游任务上的实验表明,对于NER任务改进了+0.8%,对于文本分类改进了+8.8%,展示了我们方法在移动NLP应用中的通用性及有效性。
https://arxiv.org/abs/2510.07566
Hierarchical text classification (HTC) assigns documents to multiple levels of a pre-defined taxonomy. Automated patent subject classification represents one of the hardest HTC scenarios because of domain knowledge difficulty and a huge number of labels. Prior approaches only output a flat label set, which offers little insight into the reason behind predictions. Therefore, we propose Reasoning for Hierarchical Classification (RHC), a novel framework that reformulates HTC as a step-by-step reasoning task to sequentially deduce hierarchical labels. RHC trains large language models (LLMs) in two stages: a cold-start stage that aligns outputs with chain-of-thought (CoT) reasoning format and a reinforcement learning (RL) stage to enhance multi-step reasoning ability. RHC demonstrates four advantages in our experiments. (1) Effectiveness: RHC surpasses previous baselines and outperforms the supervised fine-tuning counterparts by approximately 3% in accuracy and macro F1. (2) Explainability: RHC produces natural-language justifications before prediction to facilitate human inspection. (3) Scalability: RHC scales favorably with model size with larger gains compared to standard fine-tuning. (4) Applicability: Beyond patents, we further demonstrate that RHC achieves state-of-the-art performance on other widely used HTC benchmarks, which highlights its broad applicability.
层次文本分类(HTC)将文档分配到预定义的分类体系结构中的多个层级。自动化专利主题分类代表了最复杂的HTC场景之一,因为其需要专业的领域知识,并且标签数量巨大。之前的方法只输出扁平化的标签集合,这无法提供预测背后的理由。因此,我们提出了一种新的框架——用于层次分类的理由推理(RHC),它将HTC重新表述为一个逐步推理任务,以顺序推导出层级标签。RHC通过两个阶段训练大型语言模型(LLM):第一阶段是对齐链式思维(CoT)推理格式的冷启动阶段;第二阶段是增强多步推理能力的强化学习(RL)阶段。 在我们的实验中,RHC展示了四个优势: 1. **有效性**:与之前的基准方法相比,RHC的表现更佳,并且其监督微调版本的准确性提高了大约3%。 2. **解释性**:在预测前生成自然语言的理由,有助于人工审核和理解模型的决策过程。 3. **可扩展性**:随着模型大小的增长,RHC表现出更好的性能提升效果,相比标准微调方法有更大的收益。 4. **适用性**:除了专利分类之外,我们在其他广泛使用的HTC基准上也展示了RHC能达到最先进的表现水平,这突显了其广泛的适用性和通用性。
https://arxiv.org/abs/2510.07167
Quantum compute is scaling fast, from cloud QPUs to high throughput GPU simulators, making it timely to prototype quantum NLP beyond toy tasks. However, devices remain qubit limited and depth limited, training can be unstable, and classical attention is compute and memory heavy. This motivates compact, phase aware quantum token mixers that stabilize amplitudes and scale to long sequences. We present CLAQS, a compact, fully quantum token mixer for text classification that jointly learns complex-valued mixing and nonlinear transformations within a unified quantum circuit. To enable stable end-to-end optimization, we apply l1 normalization to regulate amplitude scaling and introduce a two-stage parameterized quantum architecture that decouples shared token embeddings from a window-level quantum feed-forward module. Operating under a sliding-window regime with document-level aggregation, CLAQS requires only eight data qubits and shallow circuits, yet achieves 91.64% accuracy on SST-2 and 87.08% on IMDB, outperforming both classical Transformer baselines and strong hybrid quantum-classical counterparts.
量子计算的发展速度非常快,从云端的量子处理单元(QPUs)到高吞吐量的GPU模拟器,这使得在玩具任务之外开发量子自然语言处理原型变得尤为紧迫。然而,设备仍然受到量子比特数量和深度的限制,训练过程可能不稳定,并且经典的注意力机制对计算资源和内存消耗较大。因此,设计紧凑、相位感知型的量子令牌混合器来稳定振幅并扩展到长序列的任务显得尤为重要。 我们提出了CLAQS(Compact Layer-Aware Quantum Sequence),这是一种用于文本分类任务的小巧而完全量子化的令牌混合器,在统一的量子电路内同时学习复值混合和非线性变换。为了实现端到端优化的稳定性,我们在模型中应用了L1正则化来调控振幅缩放,并引入了一种两阶段参数化量子架构,这种架构将共享令牌嵌入与窗口级的量子前馈模块分离。 在基于滑动窗口机制和文档级别聚合操作的前提下,CLAQS仅需要8个数据量子比特和浅层电路结构。尽管如此,在SST-2数据集上,它依然能够达到91.64%的准确率;而在IMDB数据集上,则达到了87.08%的准确率,这些结果优于经典的Transformer基线模型以及强大的混合型量子—经典模型。
https://arxiv.org/abs/2510.06532
Deep transformer models excel at multi-label text classification but often violate domain logic that experts consider essential, an issue of particular concern in safety-critical applications. We propose a hybrid neuro-symbolic framework that integrates Answer Set Programming (ASP) with transformer-based learning on the Aviation Safety Reporting System (ASRS) corpus. Domain knowledge is formalized as weighted ASP rules and validated using the Clingo solver. These rules are incorporated in two complementary ways: (i) as rule-based data augmentation, generating logically consistent synthetic samples that improve label diversity and coverage; and (ii) as a fuzzy-logic regularizer, enforcing rule satisfaction in a differentiable form during fine-tuning. This design preserves the interpretability of symbolic reasoning while leveraging the scalability of deep neural architectures. We further tune per-class thresholds and report both standard classification metrics and logic-consistency rates. Compared to a strong Binary Cross-Entropy (BCE) baseline, our approach improves micro- and macro-F1 scores and achieves up to an 86% reduction in rule violations on the ASRS test set. To the best of our knowledge, this constitutes the first large-scale neuro-symbolic application to ASRS reports that unifies ASP-based reasoning, rule-driven augmentation, and differentiable transformer training for trustworthy, safety-critical NLP.
深度变换器模型在多标签文本分类方面表现出色,但常常违反专家认为至关重要的领域逻辑,这对安全关键型应用尤为令人担忧。我们提出了一种将回答集编程(ASP)与基于变压器的学习相结合的混合神经符号框架,并应用于航空安全管理报告系统(ASRS)语料库。领域知识被形式化为加权ASP规则并通过Clingo求解器进行验证。这些规则以两种互补方式整合:(i) 作为基于规则的数据增强,生成逻辑一致的人工合成样本,增加标签多样性和覆盖率;以及 (ii) 作为一种模糊逻辑正则项,在微调过程中以可微分形式强制执行规则满足性。这种设计在利用深度神经架构的可扩展性的同时,保留了符号推理的解释能力。我们进一步调整每个类别的阈值,并报告标准分类指标和逻辑一致性率。与强大的二元交叉熵(BCE)基准相比,我们的方法提高了微平均F1分数和宏平均F1分数,在ASRS测试集上实现了高达86%的规则违反减少。据我们所知,这是第一个将ASP推理、基于规则的数据增强以及可微分变换器训练相结合的大规模神经符号应用于ASRS报告的安全关键型NLP应用。
https://arxiv.org/abs/2510.05451
Identifying cultural capital (CC) themes in student reflections can offer valuable insights that help foster equitable learning environments in classrooms. However, themes such as aspirational goals or family support are often woven into narratives, rather than appearing as direct keywords. This makes them difficult to detect for standard NLP models that process sentences in isolation. The core challenge stems from a lack of awareness, as standard models are pre-trained on general corpora, leaving them blind to the domain-specific language and narrative context inherent to the data. To address this, we introduce AWARE, a framework that systematically attempts to improve a transformer model's awareness for this nuanced task. AWARE has three core components: 1) Domain Awareness, adapting the model's vocabulary to the linguistic style of student reflections; 2) Context Awareness, generating sentence embeddings that are aware of the full essay context; and 3) Class Overlap Awareness, employing a multi-label strategy to recognize the coexistence of themes in a single sentence. Our results show that by making the model explicitly aware of the properties of the input, AWARE outperforms a strong baseline by 2.1 percentage points in Macro-F1 and shows considerable improvements across all themes. This work provides a robust and generalizable methodology for any text classification task in which meaning depends on the context of the narrative.
在学生反思中识别文化资本(CC)主题可以为促进课堂中的公平学习环境提供有价值的见解。然而,诸如抱负目标或家庭支持等主题通常交织于叙述之中,并未以直接关键词的形式出现,这使得标准自然语言处理(NLP)模型难以检测这些主题,因为它们在处理句子时孤立地工作。这一核心挑战源于现有模型缺乏领域意识:标准模型是基于通用语料库预训练的,因而无法识别数据中特有的领域特定语言和叙述上下文。 为了解决这个问题,我们提出了AWARE框架,该框架系统性地尝试提高转换器模型在这项细致任务中的认知能力。AWARE包括三个核心组成部分: 1. **领域意识**:调整模型词汇以适应学生反思的语体风格。 2. **情境意识**:生成能够反映整篇文章上下文的句子嵌入。 3. **类别重叠意识**:采用多标签策略,识别单个句子中主题共存的情况。 我们的实验结果表明,通过使模型明确地了解输入的特点,AWARE在宏平均F1分数上超越了强大的基线模型2.1个百分点,并且在所有主题上都取得了显著的改进。这项工作为任何文本分类任务提供了一种稳健且通用的方法论,其中意义取决于叙述上下文。
https://arxiv.org/abs/2510.04983
Bayesian Optimization (BO) has been widely used to efficiently optimize expensive black-box functions with limited evaluations. In this paper, we investigate the use of BO for prompt engineering to enhance text classification with Large Language Models (LLMs). We employ an LLM-powered Gaussian Process (GP) as the surrogate model to estimate the performance of different prompt candidates. These candidates are generated by an LLM through the expansion of a set of seed prompts and are subsequently evaluated using an Upper Confidence Bound (UCB) acquisition function in conjunction with the GP posterior. The optimization process iteratively refines the prompts based on a subset of the data, aiming to improve classification accuracy while reducing the number of API calls by leveraging the prediction uncertainty of the LLM-based GP. The proposed BO-LLM algorithm is evaluated on two datasets, and its advantages are discussed in detail in this paper.
贝叶斯优化(BO)已被广泛用于高效地优化昂贵的黑盒函数,同时限制评估次数。本文研究了使用贝叶斯优化进行提示工程以增强大型语言模型(LLMs)在文本分类中的效果。我们采用由LLM驱动的高斯过程(GP)作为代理模型来估计不同提示候选方案的表现。这些候选方案通过扩展一组种子提示并通过LLM生成,随后利用与GP后验概率结合的上置信界(UCB)获取函数进行评估。优化过程基于数据子集迭代地改进提示,旨在提高分类准确性的同时减少API调用次数,并通过LLM基GP预测不确定性实现这一目标。本文提出的BO-LLM算法在两个数据集上进行了评估,其优点也在文中详细讨论。
https://arxiv.org/abs/2510.04384
Pre-trained language models have achieved remarkable success across a wide range of natural language processing (NLP) tasks, particularly when fine-tuned on large, domain-relevant datasets. However, they remain vulnerable to backdoor attacks, where adversaries embed malicious behaviors using trigger patterns in the training data. These triggers remain dormant during normal usage, but, when activated, can cause targeted misclassifications. In this work, we investigate the internal behavior of backdoored pre-trained encoder-based language models, focusing on the consistent shift in attention and gradient attribution when processing poisoned inputs; where the trigger token dominates both attention and gradient signals, overriding the surrounding context. We propose an inference-time defense that constructs anomaly scores by combining token-level attention and gradient information. Extensive experiments on text classification tasks across diverse backdoor attack scenarios demonstrate that our method significantly reduces attack success rates compared to existing baselines. Furthermore, we provide an interpretability-driven analysis of the scoring mechanism, shedding light on trigger localization and the robustness of the proposed defense.
预训练语言模型在广泛的自然语言处理(NLP)任务中取得了显著的成功,特别是在大规模、领域相关的数据集上进行微调时。然而,它们仍然容易受到后门攻击的威胁,在这种攻击中,对手会在训练数据中嵌入恶意行为使用触发模式。这些触发器在正常使用期间处于休眠状态,但在被激活时会导致目标误分类。 在这项工作中,我们调查了带有后门预训练编码器语言模型内部的行为,重点关注处理受污染输入时注意力和梯度归因的一致性变化;在这种情况下,触发令牌主导了注意力和梯度信号,覆盖了周围的上下文。我们提出了一种在推理时间进行防御的方法,通过结合标记级别的注意力和梯度信息来构建异常分数。我们在各种后门攻击场景下的文本分类任务中进行了广泛的实验,结果表明我们的方法显著降低了现有基准的攻击成功率。 此外,我们提供了对评分机制的可解释性驱动分析,揭示了触发器定位和所提议防御措施鲁棒性的新见解。
https://arxiv.org/abs/2510.04347
Mixture-of-Experts (MoE) layers scale transformers by routing tokens to a sparse subset of feed-forward experts. Token-level routing, however, assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialization. We introduce SliceMoE, an architecture that routes contiguous slices of a token's hidden vector. A d-dimensional embedding is partitioned into S slices, and for each slice, a lightweight shared router predicts the top-k experts. Experts operate on their assigned slices independently, and outputs are reassembled, maintaining per-token FLOP efficiency. Because slices from different tokens interleave within an expert, utilization is naturally smoother. We propose a slice-level capacity loss, cross-slice dropout, and efficient fused batched GEMM kernels. Experiments on WikiText-103 language modeling, WMT En-De translation, and three text-classification datasets show SliceMoE attains up to 1.7x faster inference than dense baselines, 12 to 18 percent lower perplexity than parameter-matched token-MoE, and improved expert balance, with interpretable expertise over syntactic versus semantic subspaces.
混合专家(Mixture-of-Experts,MoE)层通过将令牌路由到一组稀疏的前馈专家子集来扩展转换器模型。然而,基于令牌级别的路由为每个专家分配了一个完整的语义频谱,这会导致容量瓶颈、负载均衡问题以及专业化的限制。为了应对这些问题,我们引入了SliceMoE架构,该架构根据连续的片段将令牌隐藏向量路由到不同的专家。 在SliceMoE中,一个d维嵌入被划分为S个片段(slices)。对于每一个片段,一个轻量级共享路由器预测出前k个专家。每个专家独立地操作它们分配到的片段,并且输出会被重新组装起来以保持每令牌的计算效率不变。由于来自不同令牌的片段在一个专家内交错进行处理,因此利用率会更加平滑。 我们提出了一种基于切片级别的容量损失、跨切片dropout策略以及高效的批矩阵乘法(GEMM)核来改进模型性能和效率。实验结果表明,在WikiText-103语言建模任务、WMT英德翻译任务及三个文本分类数据集上,与密集基线相比,SliceMoE实现了高达1.7倍的推理加速;与参数匹配的令牌MoE相比,困惑度降低了12%至18%,同时改进了专家间的平衡性,并且使得每个专家专注于语法或语义子空间中更具可解释性的专业知识。
https://arxiv.org/abs/2510.04286
Softmax Attention has a quadratic time complexity, which becomes prohibitive to run at long contexts, even with highly optimized GPU kernels. For example, FlashAttention (an exact, GPU-optimized implementation of Softmax Attention) cannot complete a single forward-backward pass of a multi-head attention layer once the context exceeds ~4 million tokens on an NVIDIA GH200 (96 GB). We introduce RACE Attention, a kernel-inspired alternative to Softmax Attention that is linear in sequence length and embedding dimension. RACE Attention replaces the exponential kernel with a sharpened angular (cosine) similarity, and approximates attention outputs via randomized projections and soft Locality-Sensitive Hashing (LSH). Across language modeling, masked language modeling, and text classification, RACE Attention matches the accuracy of strong baselines while reducing runtime and memory. In a controlled scale test, it processes up to 12 million tokens during a single forward-backward pass on an NVIDIA GH200 GPU and 75 million tokens on an Intel Xeon Gold 5220R CPU, well beyond the practical limits of the current state-of-the-art attention implementations. RACE Attention thus offers a practical, theoretically grounded mechanism for outrageously long context windows on today's hardware. We hope that it gets adopted in practice.
Softmax注意力机制的时间复杂度为二次方,这在处理长上下文时变得非常不可行,即使使用了高度优化的GPU内核也是如此。例如,FlashAttention(一种精确且经过GPU优化的Softmax Attention实现)在NVIDIA GH200 GPU上一旦上下文超过约4百万令牌,就无法完成多头注意力层的一次前向-反向传递。 我们引入了一种名为RACE Attention的新方法,这是一种受内核启发的替代方案,它的时间复杂度与序列长度和嵌入维度呈线性关系。RACE Attention通过使用锐化的余弦相似度(即角相似度)来替换指数核,并且通过随机投影和软局部敏感哈希(LSH)近似注意力输出。 在语言建模、遮蔽语言建模以及文本分类等多种任务中,RACE Attention能够在减少运行时间和内存消耗的同时与强大的基准模型达到相同的准确性。在一个受控的规模测试中,在NVIDIA GH200 GPU上处理多达1.2亿个令牌,在Intel Xeon Gold 5220R CPU上处理多达7.5亿个令牌的一次前向-反向传递,远远超过了当前最先进的注意力机制实现的实际限制。 因此,RACE Attention为当今硬件上的极长上下文窗口提供了理论上合理且实用的解决方案。我们希望它能在实践中被采用。
https://arxiv.org/abs/2510.04008
Traditional object detection models are typically trained on a fixed set of classes, limiting their flexibility and making it costly to incorporate new categories. Open-vocabulary object detection addresses this limitation by enabling models to identify unseen classes without explicit training. Leveraging pretrained models contrastively trained on abundantly available ground-view image-text classification pairs provides a strong foundation for open-vocabulary object detection in aerial imagery. Domain shifts, viewpoint variations, and extreme scale differences make direct knowledge transfer across domains ineffective, requiring specialized adaptation strategies. In this paper, we propose a novel framework for adapting open-vocabulary representations from ground-view images to solve object detection in aerial imagery through structured domain alignment. The method introduces contrastive image-to-image alignment to enhance the similarity between aerial and ground-view embeddings and employs multi-instance vocabulary associations to align aerial images with text embeddings. Extensive experiments on the xView, DOTAv2, VisDrone, DIOR, and HRRSD datasets are used to validate our approach. Our open-vocabulary model achieves improvements of +6.32 mAP on DOTAv2, +4.16 mAP on VisDrone (Images), and +3.46 mAP on HRRSD in the zero-shot setting when compared to finetuned closed-vocabulary dataset-specific model performance, thus paving the way for more flexible and scalable object detection systems in aerial applications.
传统的对象检测模型通常在固定的一组类别上进行训练,这限制了其灵活性,并且增加了将新类别纳入系统的成本。开放词汇对象检测通过使模型能够在没有显式训练的情况下识别未见过的类来解决这一限制。利用预先训练好的模型,这些模型是在大量可用的地平视图像-文本分类对上对比性训练得到的,为在航拍图像中进行开放词汇对象检测提供了坚实的基础。 然而,领域变化、视角差异和极端尺度不同使得直接跨域知识迁移无效,需要专门的适应策略。在这篇论文中,我们提出了一种新颖的框架,用于将地平视图图像中的开放词汇表示适应到解决航拍图像中的对象检测问题,通过结构化的领域对齐来实现这一目标。该方法引入了对比性的图像对齐以增强航拍与地平视图嵌入之间的相似性,并采用多实例词汇关联来使航拍图像与文本嵌入对齐。 我们在xView、DOTAv2、VisDrone、DIOR和HRRSD数据集上进行了广泛的实验,用以验证我们的方法。在零样本设置中,我们的开放词汇模型相比经过微调的封闭词汇特定数据集模型,在DOTAv2上的mAP提升了+6.32%,在VisDrone(图像)上的mAP提升了+4.16%,在HRRSD上提升为+3.46%。这表明了更灵活和可扩展的对象检测系统在航拍应用中的潜力。
https://arxiv.org/abs/2510.03858