Current meta-learning methods are constrained to narrow task distributions with fixed feature and label spaces, limiting applicability. Moreover, the current meta-learning literature uses key terms like "universal" and "general-purpose" inconsistently and lacks precise definitions, hindering comparability. We introduce a theoretical framework for meta-learning which formally defines practical universality and introduces a distinction between algorithm-explicit and algorithm-implicit learning, providing a principled vocabulary for reasoning about universal meta-learning methods. Guided by this framework, we present TAIL, a transformer-based algorithm-implicit meta-learner that functions across tasks with varying domains, modalities, and label configurations. TAIL features three innovations over prior transformer-based meta-learners: random projections for cross-modal feature encoding, random injection label embeddings that extrapolate to larger label spaces, and efficient inline query processing. TAIL achieves state-of-the-art performance on standard few-shot benchmarks while generalizing to unseen domains. Unlike other meta-learning methods, it also generalizes to unseen modalities, solving text classification tasks despite training exclusively on images, handles tasks with up to 20$\times$ more classes than seen during training, and provides orders-of-magnitude computational savings over prior transformer-based approaches.
https://arxiv.org/abs/2602.14761
Recent advances in Natural Language Processing have been predominantly driven by transformer-based architectures, which rely heavily on self-attention mechanisms to model relationships between tokens in a sequence. Similarly, the field of Quantum Natural Language Processing, which seeks to leverage quantum principles to address challenges in language understanding and generation tasks, has seen the recent development of quantum self-attention mechanisms. We propose a classical quantum-inspired self-attention (QISA) mechanism and integrate it into the full autoregressive language modeling pipeline of GPT-1. To the best of our knowledge, this is the first integration of this kind, as previous quantum self-attention mechanisms have been primarily tested on text classification. In our experiments, QISA achieves better performance when compared to standard self-attention on the metrics character error rate ($15.5\times$ better), word error rate ($4.7 \times $) and cross-entropy loss ($13 \times$). This is achieved while only requiring a $ 2.6\times$ longer inference time.
最近的自然语言处理(NLP)进展主要由基于变压器架构的技术推动,这些技术依赖于自注意力机制来建模序列中词汇单元之间的关系。同样,在量子自然语言处理领域——旨在利用量子原理解决语言理解和生成任务中的挑战——人们近期开发出了量子自注意力机制。我们提出了一种受量子启发的经典自注意力(QISA)机制,并将其集成到了GPT-1的完整自回归语言建模管道中。据我们所知,这是此类集成的首次尝试,因为之前的量子自注意力机制主要在文本分类任务上进行了测试。在我们的实验中,QISA相较于标准自注意力机制,在字符错误率(低15.5倍)、单词错误率(低4.7倍)和交叉熵损失(低13倍)这些指标上取得了更好的表现。同时,这仅需要2.6倍的推理时间。 翻译为中文如下: 最近在自然语言处理方面的进展主要由基于变换器架构的技术推动,这些技术依赖于自注意力机制来建模序列中词汇单元之间的关系。同样,在量子自然语言处理领域——旨在利用量子原理解决语言理解和生成任务中的挑战——人们近期开发出了量子自注意力机制。我们提出了一种受量子启发的经典自注意力(QISA)机制,并将其集成到了GPT-1的完整自回归语言建模管道中。据我们所知,这是首次此类集成,因为之前的量子自注意力机制主要在文本分类任务上进行了测试。在我们的实验中,与标准自注意力相比,QISA在字符错误率(降低15.5倍)、单词错误率(降低4.7倍)和交叉熵损失(降低13倍)这些指标上取得了更好的表现,并且只需要增加2.6倍的推理时间。
https://arxiv.org/abs/2603.03318
Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.
在全球超过7000种语言中,商用语言识别(LID)系统仅能可靠地识别几百种语言的书面形式。研究级别的系统在特定情况下可以扩大这一覆盖范围,但对于大多数语言而言,覆盖率仍然支离破碎或根本不存在。本文认为这种情况主要是自我限制的结果。特别是,它源于将LID视为脱离上下文的文本分类这种持久的观点,这掩盖了先验概率估计的核心作用,并且受到倾向于全球性、固定先验模型的制度激励所强化。我们认为,要提高尾部语言(即使用较少的语言)的覆盖率,需要重新审视LID作为一个路由问题的本质,并开发出有原则的方法来整合环境线索,使各种语言在局部环境中变得合理可信。
https://arxiv.org/abs/2602.08951
Large Language Models (LLMs) have improved substantially alignment, yet their behavior remains highly sensitive to prompt phrasing. This brittleness has motivated automated prompt engineering, but most existing methods (i) require a task-specific training set, (ii) rely on expensive iterative optimization to produce a single dataset-level prompt, and (iii) must be rerun from scratch for each new task. We introduce TATRA, a dataset-free prompting method that constructs instance-specific few-shot prompts by synthesizing on-the-fly examples to accompany a user-provided instruction. TATRA requires no labeled training data and avoids task-specific optimization loops, while retaining the benefits of demonstration-based prompting. Across standard text classification benchmarks, TATRA matches or improves over strong prompt-optimization baselines that depend on training data and extensive search. On mathematical reasoning benchmarks, TATRA achieves state-of-the-art performance on GSM8K and DeepMath, outperforming methods that explicitly optimize prompts on those tasks. Our results suggest that per-instance construction of effective in-context examples is more important than running long, expensive optimization loops to produce a single prompt per task. We will make all code publicly available upon acceptance of the paper. Code is available at this https URL
大型语言模型(LLMs)在对齐方面有了显著的改进,但它们的行为仍然高度依赖于提示语句的表述。这种脆弱性促使了自动化提示工程的发展,然而大多数现有的方法需要特定任务的训练集、依靠昂贵的迭代优化来生成单一的数据级提示,并且每次遇到新的任务时都需要从头开始重新运行。 我们引入了一种名为TATRA的方法,这是一种不依赖于数据集的提示技术,它通过实时合成示例以伴随用户提供的指令来构建针对每个实例的少量样本提示。TATRA不需要标注的训练数据并且避免了特定任务的优化循环,同时保留了基于演示的提示方法的优点。 在标准文本分类基准测试中,与依赖于训练数据和大量搜索的强大提示优化基线相比,TATRA能够达到或超越它们的表现水平。在数学推理基准上,TATRA在GSM8K和DeepMath任务上实现了最先进的性能,超过了那些专门在这类任务上进行提示优化的方法。 我们的研究结果表明,针对每个实例构建有效的上下文示例比为单一的任务运行漫长且昂贵的优化循环更为重要。一旦论文被接受,我们将公开所有代码。代码可以在提供的链接处获取(假设原文中的链接可以正常使用)。
https://arxiv.org/abs/2603.03298
Large language models (LLMs) such as GPT-4o and Claude Sonnet 4.5 have demonstrated strong capabilities in open-ended reasoning and generative language tasks, leading to their widespread adoption across a broad range of NLP applications. However, for structured text classification problems with fixed label spaces, model selection is often driven by predictive performance alone, overlooking operational constraints encountered in production systems. In this work, we present a systematic comparison of two contrasting paradigms for text classification: zero- and few-shot prompt-based large language models, and fully fine-tuned encoder-only architectures. We evaluate these approaches across four canonical benchmarks (IMDB, SST-2, AG News, and DBPedia), measuring predictive quality (macro F1), inference latency, and monetary cost. We frame model evaluation as a multi-objective decision problem and analyze trade-offs using Pareto frontier projections and a parameterized utility function reflecting different deployment regimes. Our results show that fine-tuned encoder-based models from the BERT family achieve competitive, and often superior, classification performance while operating at one to two orders of magnitude lower cost and latency compared to zero- and few-shot LLM prompting. Overall, our findings suggest that indiscriminate use of large language models for standard text classification workloads can lead to suboptimal system-level outcomes. Instead, fine-tuned encoders emerge as robust and efficient components for structured NLP pipelines, while LLMs are better positioned as complementary elements within hybrid architectures. We release all code, datasets, and evaluation protocols to support reproducibility and cost-aware NLP system design.
大型语言模型(LLMs)如GPT-4o和Claude Sonnet 4.5在开放性推理和生成语言任务中展现了强大的能力,因此这些模型被广泛应用于各种自然语言处理(NLP)应用。然而,在结构化的文本分类问题中,这些问题具有固定的标签空间时,模型的选择往往只关注预测性能,而忽略了生产系统中的操作限制。在这项工作中,我们提出了两种截然不同的文本分类方法的系统比较:零样本和少量样本文本提示的大语言模型与完全微调过的编码器架构。我们在四个标准基准(IMDB、SST-2、AG News 和 DBPedia)上评估了这些方法,并测量了预测质量(宏F1分数)、推理延迟以及经济成本。我们将模型评估框架化为一个多目标决策问题,并通过帕累托前沿投影和反映不同部署方案的参数化效用函数来分析权衡。我们的结果表明,基于BERT家族的完全微调编码器模型在分类性能上具有竞争力甚至更优,同时其运行成本和延迟仅为零样本或少量样本文本提示的大语言模型的一到两个数量级。 总体而言,我们的研究发现认为,在标准文本分类工作负载中无差别地使用大型语言模型会导致次优的系统级结果。相反,微调过的编码器作为结构化NLP管道中的稳健且高效的组件而出现,而LLM则更适合在混合架构中作为补充元素。我们发布了所有代码、数据集和评估协议以支持可重复性以及成本意识型NLP系统的构建。
https://arxiv.org/abs/2602.06370
Medical vision-language models show strong potential for joint reasoning over medical images and clinical text, but their performance often degrades under domain shift caused by variations in imaging devices, acquisition protocols, and reporting styles. Existing multi-modal pre-training methods largely overlook robustness, treating it as a downstream adaptation problem. In this work, we propose Robust Multi-Modal Masked Reconstruction (Robust-MMR), a self-supervised pre-training framework that explicitly incorporates robustness objectives into masked vision-language learning. Robust-MMR integrates asymmetric perturbation-aware masking, domain-consistency regularization, and modality-resilience constraints to encourage domain-invariant representations. We evaluate Robust-MMR on multiple medical vision-language benchmarks, including medical visual question answering (VQA-RAD, SLAKE, VQA-2019), cross-domain image-text classification (MELINDA), and robust image-caption retrieval (ROCO). Robust-MMR achieves 78.9% cross-domain accuracy on VQA-RAD, outperforming the strongest baseline by 3.8 percentage points, and reaches 74.6% and 77.0% accuracy on SLAKE and VQA-2019, respectively. Under perturbed evaluation, Robust-MMR improves VQA-RAD accuracy from 69.1% to 75.6%. For image-text classification, cross-domain MELINDA accuracy increases from 70.3% to 75.2%, while retrieval experiments show a reduction in mean rank degradation from over 16 to 4.1 under perturbation. Qualitative results further demonstrate improved clinical reasoning for disease detection and structural abnormality assessment. These findings show that explicitly modeling robustness during pre-training leads to more reliable and transferable medical vision-language representations for real-world deployment.
医学视觉-语言模型在对医疗图像和临床文本进行联合推理方面展示了强大的潜力,但其性能往往因成像设备、采集协议和报告风格的差异而产生的领域变化(domain shift)而下降。现有的多模态预训练方法很大程度上忽视了鲁棒性问题,并将其视为下游适应任务。在此项研究中,我们提出了Robust Multi-Modal Masked Reconstruction (Robust-MMR),这是一种自监督预训练框架,它明确地将鲁棒性的目标纳入到了遮蔽视觉-语言学习之中。Robust-MMR集成了不对称扰动感知的屏蔽、领域一致性正则化和模态韧性约束以鼓励领域的不变表示。我们对多个医学视觉-语言基准进行了评估,包括医疗视觉问答(VQA-RAD, SLAKE, VQA-2019)、跨域图像文本分类(MELINDA)以及鲁棒的图像标题检索(ROCO)。在VQA-RAD上,Robust-MMR实现了78.9%的跨领域准确率,比最强基线高出了3.8个百分点,并分别达到了SLAKE和VQA-2019上的74.6%和77.0%准确率。在扰动评估下,Robust-MMR将VQA-RAD的准确性从69.1%提高到了75.6%。对于图像文本分类,在跨领域的MELINDA上精度从70.3%提升到75.2%,而在检索实验中显示了均位序退化率在扰动下的减少,从超过16降至4.1。定性结果进一步展示了改进的临床推理能力,用于疾病检测和结构异常评估。这些发现表明,在预训练期间显式地建模鲁棒性可以为真实世界的部署生成更可靠且可转移的医学视觉-语言表示。
https://arxiv.org/abs/2602.17689
Large language models (LLMs) are widely used as zero-shot and few-shot classifiers, where task behaviour is largely controlled through prompting. A growing number of works have observed that LLMs are sensitive to prompt variations, with small changes leading to large changes in performance. However, in many cases, the investigation of sensitivity is performed using underspecified prompts that provide minimal task instructions and weakly constrain the model's output space. In this work, we argue that a significant portion of the observed prompt sensitivity can be attributed to prompt underspecification. We systematically study and compare the sensitivity of underspecified prompts and prompts that provide specific instructions. Utilising performance analysis, logit analysis, and linear probing, we find that underspecified prompts exhibit higher performance variance and lower logit values for relevant tokens, while instruction-prompts suffer less from such problems. However, linear probing analysis suggests that the effects of prompt underspecification have only a marginal impact on the internal LLM representations, instead emerging in the final layers. Overall, our findings highlight the need for more rigour when investigating and mitigating prompt sensitivity.
大型语言模型(LLMs)被广泛用作零样本和少量样本分类器,其任务行为主要通过提示来控制。越来越多的研究发现,LLM 对提示的变化非常敏感,即使是微小的改变也可能导致性能上的大幅波动。然而,在许多情况下,对这种敏感性的调查是使用提供最少任务说明且弱化了模型输出空间约束的不明确提示来进行的。在本工作中,我们认为观察到的大部分提示敏感性可以归因于提示的不明确性。我们系统地研究并比较了不明确提示与提供具体指令的提示之间的敏感度差异。通过性能分析、对数几率值分析以及线性探测,我们发现不明确提示表现出更高的性能波动和更低的相关词令牌的对数值,而指令提示则较少受到这些问题的影响。然而,线性探测分析表明,提示不明确性的效果只在模型内部表示的最终层中产生边际影响,而不是影响整体表示结构。总体而言,我们的研究结果强调了在调查和缓解提示敏感性时需要更加严谨的态度。
https://arxiv.org/abs/2602.04297
Finding effective prompts for language models (LMs) is critical yet notoriously difficult: the prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based prompt optimizers often rely on on-policy updates and a meta-prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic prompt optimization framework that casts prompt search as a posterior inference problem over latent prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic Memory Update (DMU), a training-free mechanism that updates the meta-prompt by injecting both (i) diverse prompts from a replay buffer and (ii) top-performing prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete prompt optimization baselines.
寻找有效的提示对于语言模型(LM)来说是至关重要的,但同时也非常困难:提示空间在组合上庞大无比,由于目标语言模型的评估成本高昂,因此奖励也极为稀疏。然而,现有的基于强化学习(RL)的提示优化器通常依赖于在线策略更新和从固定分布中抽取的元提示,这导致了较差的样本效率。 我们提出了GFlowPO,这是一种概率性的提示优化框架,它将提示搜索视为一种通过元提示引导的语言模型先验进行后验推理的问题。在第一步中,我们使用基于离线策略的目标生成流网络(Generative Flow Network, GFlowNet)目标训练了一个轻量级的提示语言模型,并采用了基于回放的训练策略来重用过去的提示评估以实现高效的样本探索。第二步中,我们引入了动态内存更新(Dynamic Memory Update, DMU),这是一种无需训练的方法,它通过注入来自回放缓冲区的多样化的提示以及来自优先队列的小型高绩效提示,逐步将搜索过程集中在高奖励区域。 在少量样本文本分类、指令诱导基准测试和问答任务中,GFlowPO始终优于最近提出的离散提示优化基线方法。
https://arxiv.org/abs/2602.03358
Natural language explanations play a critical role in establishing trust and acceptance of automated vehicles (AVs), yet existing approaches lack systematic frameworks for analysing how humans linguistically construct driving rationales across diverse scenarios. This paper introduces X-Blocks (eXplanation Blocks), a hierarchical analytical framework that identifies the linguistic building blocks of natural language explanations for AVs at three levels: context, syntax, and lexicon. At the context level, we propose RACE (Reasoning-Aligned Classification of Explanations), a multi-LLM ensemble framework that combines Chain-of-Thought reasoning with self-consistency mechanisms to robustly classify explanations into 32 scenario-aware categories. Applied to human-authored explanations from the Berkeley DeepDrive-X dataset, RACE achieves 91.45 percent accuracy and a Cohens kappa of 0.91 against cases with human annotator agreement, indicating near-human reliability for context classification. At the lexical level, log-odds analysis with informative Dirichlet priors reveals context-specific vocabulary patterns that distinguish driving scenarios. At the syntactic level, dependency parsing and template extraction show that explanations draw from a limited repertoire of reusable grammar families, with systematic variation in predicate types and causal constructions across contexts. The X-Blocks framework is dataset-agnostic and task-independent, offering broad applicability to other automated driving datasets and safety-critical domains. Overall, our findings provide evidence-based linguistic design principles for generating scenario-aware explanations that support transparency, user trust, and cognitive accessibility in automated driving systems.
https://arxiv.org/abs/2602.13248
One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.
https://arxiv.org/abs/2601.05219
Automated peer review has evolved from simple text classification to structured feedback generation. However, current state-of-the-art systems still struggle with "surface-level" critiques: they excel at summarizing content but often fail to accurately assess novelty and significance or identify deep methodological flaws because they evaluate papers in a vacuum, lacking the external context a human expert possesses. In this paper, we introduce ScholarPeer, a search-enabled multi-agent framework designed to emulate the cognitive processes of a senior researcher. ScholarPeer employs a dual-stream process of context acquisition and active verification. It dynamically constructs a domain narrative using a historian agent, identifies missing comparisons via a baseline scout, and verifies claims through a multi-aspect Q&A engine, grounding the critique in live web-scale literature. We evaluate ScholarPeer on DeepReview-13K and the results demonstrate that ScholarPeer achieves significant win-rates against state-of-the-art approaches in side-by-side evaluations and reduces the gap to human-level diversity.
https://arxiv.org/abs/2601.22638
The increasing availability of unstructured clinical narratives in electronic health records (EHRs) has created new opportunities for automated disease characterization, cohort identification, and clinical decision support. However, modeling long, domain-specific clinical text remains challenging due to limited labeled data, severe class imbalance, and the high computational cost of adapting large pretrained language models. This study presents a GPT-based architecture for clinical text classification that adapts a pretrained decoder-only Transformer using a selective fine-tuning strategy. Rather than updating all model parameters, the majority of the GPT-2 backbone is frozen, and training is restricted to the final Transformer block, the final layer normalization, and a lightweight classification head. This approach substantially reduces the number of trainable parameters while preserving the representational capacity required to model complex clinical language. The proposed method is evaluated on radiology reports from the MIMIC-IV-Note dataset using uncertainty-aware CheXpert-style labels derived directly from report text. Experiments cover multiple problem formulations, including multi-label classification of radiographic findings, binary per-label classification under different uncertainty assumptions, and aggregate disease outcome prediction. Across varying dataset sizes, the model exhibits stable convergence behavior and strong classification performance, particularly in settings dominated by non-mention and negated findings. Overall, the results indicate that selective fine-tuning of pretrained generative language models provides an efficient and effective pathway for clinical text classification, enabling scalable adaptation to real-world EHR data while significantly reducing computational complexity.
https://arxiv.org/abs/2601.21955
Hierarchical text classification (HTC) depends on taxonomies that organize labels into structured hierarchies. However, many real-world taxonomies introduce ambiguities, such as identical leaf names under similar parent nodes, which prevent language models (LMs) from learning clear decision boundaries. In this paper, we present TaxMorph, a framework that uses large language models (LLMs) to transform entire taxonomies through operations such as renaming, merging, splitting, and reordering. Unlike prior work, our method revises the full hierarchy to better match the semantics encoded by LMs. Experiments across three HTC benchmarks show that LLM-refined taxonomies consistently outperform human-curated ones in various settings up to +2.9pp. in F1. To better understand these improvements, we compare how well LMs can assign leaf nodes to parent nodes and vice versa across human-curated and LLM-refined taxonomies. We find that human-curated taxonomies lead to more easily separable clusters in embedding space. However, the LLM-refined taxonomies align more closely with the model's actual confusion patterns during classification. In other words, even though they are harder to separate, they better reflect the model's inductive biases. These findings suggest that LLM-guided refinement creates taxonomies that are more compatible with how models learn, improving HTC performance.
https://arxiv.org/abs/2601.18375
Intent detection, a fundamental text classification task, aims to identify and label the semantics of user queries, playing a vital role in numerous business applications. Despite the dominance of deep learning techniques in this field, the internal mechanisms enabling Recurrent Neural Networks (RNNs) to solve intent detection tasks are poorly understood. In this work, we apply dynamical systems theory to analyze how RNN architectures address this problem, using both the balanced SNIPS and the imbalanced ATIS datasets. By interpreting sentences as trajectories in the hidden state space, we first show that on the balanced SNIPS dataset, the network learns an ideal solution: the state space, constrained to a low-dimensional manifold, is partitioned into distinct clusters corresponding to each intent. The application of this framework to the imbalanced ATIS dataset then reveals how this ideal geometric solution is distorted by class imbalance, causing the clusters for low-frequency intents to degrade. Our framework decouples geometric separation from readout alignment, providing a novel, mechanistic explanation for real world performance disparities. These findings provide new insights into RNN dynamics, offering a geometric interpretation of how dataset properties directly shape a network's computational solution.
https://arxiv.org/abs/2601.17156
Introduction: Clinical text classification using natural language processing (NLP) models requires adequate training data to achieve optimal performance. For that, 200-500 documents are typically annotated. The number is constrained by time and costs and lacks justification of the sample size requirements and their relationship to text vocabulary properties. Methods: Using the publicly available MIMIC-III dataset containing hospital discharge notes with ICD-9 diagnoses as labels, we employed pre-trained BERT embeddings followed by Random Forest classifiers to identify 10 randomly selected diagnoses, varying training corpus sizes from 100 to 10,000 documents, and analyzed vocabulary properties by identifying strong and noisy predictive words through Lasso logistic regression on bag-of-words embeddings. Results: Learning curves varied significantly across the 10 classification tasks despite identical preprocessing and algorithms, with 600 documents sufficient to achieve 95% of the performance attainable with 10,000 documents for all tasks. Vocabulary analysis revealed that more strong predictors and fewer noisy predictors were associated with steeper learning curves, where every 100 additional noisy words decreased accuracy by approximately 0.02 while 100 additional strong predictors increased maximum accuracy by approximately 0.04.
引言:使用自然语言处理(NLP)模型进行临床文本分类需要充足的训练数据以达到最佳性能。通常情况下,这要求对大约200到500份文档进行注释。然而,这个数字受到时间和成本的限制,并且缺乏关于样本量需求及其与文本词汇属性关系的合理解释。 方法:我们使用了公开可用的MIMIC-III数据集,该数据集中包含医院出院记录及相应的ICD-9诊断标签。通过应用预训练的BERT嵌入并随后采用随机森林分类器来识别10个随机选择的诊断,并且在从100到10,000份文档不等的训练语料库大小下进行了实验。此外,我们还利用bag-of-words(词袋)模型中的Lasso逻辑回归分析词汇属性,通过识别强预测词和噪声预测词来实现这一目标。 结果:尽管预处理过程与算法保持一致,但针对10个分类任务的学习曲线差异显著。对于所有任务而言,使用600份文档即可达到使用10,000份文档所能获得的95%性能水平。词汇分析揭示了更多的强预测词和更少的噪声预测词与陡峭的学习曲线之间的关联性:每增加100个噪声单词会使准确率降低约0.02,而每增加100个强预测词则会将最大准确度提高大约0.04。
https://arxiv.org/abs/2601.15846
Open-set learning and discovery (OSLD) is a challenging machine learning task in which samples from new (unknown) classes can appear at test time. It can be seen as a generalization of zero-shot learning, where the new classes are not known a priori, hence involving the active discovery of new classes. While zero-shot learning has been extensively studied in text classification, especially with the emergence of pre-trained language models, open-set learning and discovery is a comparatively new setup for the text domain. To this end, we introduce the first multilingual open-set learning and discovery (MOSLD) benchmark for text categorization by topic, comprising 960K data samples across 12 languages. To construct the benchmark, we (i) rearrange existing datasets and (ii) collect new data samples from the news domain. Moreover, we propose a novel framework for the OSLD task, which integrates multiple stages to continuously discover and learn new classes. We evaluate several language models, including our own, to obtain results that can be used as reference for future work. We release our benchmark at this https URL.
开放集学习与发现(OSLD)是一项具有挑战性的机器学习任务,在该任务中,测试时可能出现来自新类别的样本。这可以被视为零样本学习的一种泛化形式,其中新的类别事先未知,因此需要主动发现新类别。虽然在文本分类领域特别是在预训练语言模型出现之后对零样本学习进行了广泛的研究,但开放集学习和发现对于文本领域来说是一个相对较新的设置。 为此,我们引入了一个名为多语言开放集学习与发现(MOSLD)的基准测试,用于按主题进行文本分类,包含12种语言共96万个数据样本。为了构建该基准测试,我们(i)重新组织现有的数据集和(ii)从新闻领域收集新的数据样本。 此外,我们还提出了一种新颖的框架来处理OSLD任务,该框架整合了多个阶段以持续发现并学习新类别。我们评估了几种语言模型(包括我们自己的),从而获得未来研究可以参考的结果。 我们的基准测试可以在以下网址获取:[请在此处提供链接]。
https://arxiv.org/abs/2601.13437
(Natural Language Processing) NLP techniques such as text classification and topic discovery are very useful in many application areas including information retrieval, knowledge discovery, policy formulation, and decision-making. However, it remains a challenging problem in cases where the categories are unavailable, difficult to differentiate, or are interrelated. Social analysis with human context is an area that can benefit from text classification, as it relies substantially on text data. The focus of this paper is to enhance the classification of text according to the UN's Sustainable Development Goals (SDGs) by collecting and combining intelligence from multiple models. Combinatorial Fusion Analysis (CFA), a system fusion paradigm using a rank-score characteristic (RSC) function and cognitive diversity (CD), has been used to enhance classifier methods by combining a set of relatively good and mutually diverse classification models. We use a generative AI model to generate synthetic data for model training and then apply CFA to this classification task. The CFA technique achieves 96.73% performance, outperforming the best individual model. We compare the outcomes with those obtained from human domain experts. It is demonstrated that combining intelligence from multiple ML/AI models using CFA and getting input from human experts can, not only complement, but also enhance each other.
自然语言处理(NLP)技术,如文本分类和主题发现,在信息检索、知识发现、政策制定和决策等许多应用领域都非常有用。然而,在类别缺失、难以区分或相互关联的情况下,这仍然是一个具有挑战性的问题。社会分析中包含的人类上下文可以从文本分类中受益,因为它很大程度上依赖于文本数据。本文的重点是通过收集并整合来自多个模型的智能来增强根据联合国可持续发展目标(SDGs)对文本进行分类的能力。 组合融合分析(CFA),一种使用排名分数特性(RSC)函数和认知多样性(CD)的系统融合范式,已被用于通过结合一组相对良好且相互多样化的分类模型来改进分类方法。我们利用生成型AI模型生成合成数据以供模型训练,并将CFA应用于该分类任务。采用CFA技术达到了96.73%的表现水平,超过了单一最佳模型的成绩。 我们将结果与人类领域专家的结果进行了比较,证明了使用CFA整合多个机器学习/人工智能模型的智能并结合人类专家的意见不仅能够互补,还能相互增强彼此的能力。
https://arxiv.org/abs/2602.11168
The increasing prevalence of Large Language Models (LLMs) demands effective safeguards for their operation, particularly concerning their tendency to generate out-of-context responses. A key challenge is accurately detecting when LLMs stray from expected conversational norms, manifesting as topic shifts, factual inaccuracies, or outright hallucinations. Traditional anomaly detection struggles to directly apply within contextual semantics. This paper outlines our experiment in exploring the use of Representation Engineering (RepE) and One-Class Support Vector Machine (OCSVM) to identify subspaces within the internal states of LLMs that represent a specific context. By training OCSVM on in-context examples, we establish a robust boundary within the LLM's hidden state latent space. We evaluate out study with two open source LLMs - Llama and Qwen models in specific contextual domain. Our approach entailed identifying the optimal layers within the LLM's internal state subspaces that strongly associates with the context of interest. Our evaluation results showed promising results in identifying the subspace for a specific context. Aside from being useful in detecting in or out of context conversation threads, this research work contributes to the study of better interpreting LLMs.
大型语言模型(LLMs)的日益普及要求有效的安全措施来保障其运行,尤其是在这些模型倾向于生成不符合上下文响应的情况下。一个关键挑战是准确检测出当LLM偏离预期对话规范时的情况,这可能表现为话题转变、事实错误或直接幻觉。传统的异常检测方法难以直接应用于语境化的语义中。本文概述了我们使用表示工程(RepE)和一类支持向量机(OCSVM)探索如何识别内部状态中的子空间,这些子空间代表特定上下文的实验。 通过在具体上下文中训练OCSVM,我们在LLM的隐藏状态潜在空间内建立了一个坚实的边界。我们的研究使用了两个开源的大规模语言模型——Llama和Qwen,并针对特定的上下文领域进行了评估。我们方法的核心在于识别出LLM内部状态子空间中的最佳层,这些层与感兴趣的上下文密切相关。 评估结果显示,在确定特定上下文的空间方面取得了令人鼓舞的结果。除了在检测是否符合上下文对话线程的应用中非常有用之外,这项研究还为更好地理解和解释大型语言模型做出了贡献。
https://arxiv.org/abs/2601.12286
Bengali text classification is a Significant task in natural language processing (NLP), where text is categorized into predefined labels. Unlike English, Bengali faces challenges due to the lack of extensive annotated datasets and pre-trained language models. This study explores the effectiveness of large language models (LLMs) in classifying Bengali newspaper articles. The dataset used, obtained from Kaggle, consists of articles from Prothom Alo, a major Bangladeshi newspaper. Three instruction-tuned LLMs LLaMA 3.1 8B Instruct, LLaMA 3.2 3B Instruct, and Qwen 2.5 7B Instruct were evaluated for this task under the same classification framework. Among the evaluated models, Qwen 2.5 achieved the highest classification accuracy of 72%, showing particular strength in the "Sports" category. In comparison, LLaMA 3.1 and LLaMA 3.2 attained accuracies of 53% and 56%, respectively. The findings highlight the effectiveness of LLMs in Bengali text classification, despite the scarcity of resources for Bengali NLP. Future research will focus on exploring additional models, addressing class imbalance issues, and refining fine-tuning approaches to improve classification performance.
বাংলা টেক্সট ক্লাসিফিকেশন ন্যাচরাল ল্যাঙুয়েজ প্রসেসিং (NLP) এর একটি গুরুত্বপূর্ণ উদ্দেশ্য, যেখানে টেক্সটকে ইতিমধ্যেই ডিফাইন করা লেবেলগুলির ভিত্তিতে বিভাগ করা হয়। ইংরেজির চেয়ে, বাংলা এমন দুর্নিয়তা মোকাবেলা করে যা আইনসূচক ডেটাসেট এবং প্রিট্রেইনড ল্যাঙুয়েজ মডেলগুলির অভাবের ফলে হয়। এই গবেষণা দেখেছে উচ্চকনিষ্ঠ ল্যাঙুয়েজ মডেল (LLMs) প্রতিরোধ করে বাংলা নভিসপেপার আর্টিকেলগুলিকে ক্লাসিফিকেট করার দক্ষতা। এই গবেষণায় ব্যবহৃত ডেটাসেট, Kaggle থেকে আসা হল, যা প্রথম অলো, একটি মুখ্য বাংলাদেশি নভিল থেকে উদ্ভূত। LLaMA 3.1 8B Instruct, LLaMA 3.2 3B Instruct এবং Qwen 2.5 7B Instruct মডেলগুলি বিশেষ সমস্যার জন্য পরীক্ষা করা হয়েছে, এবং আইনসূচক লেবেলগুলির বিশ্লেষণ ফ্রেমওয়ার্কটির ভিত্তিতে। মডেলগুলির মধ্যে, Qwen 2.5 দৃঢভাবে "খেলনা" শ্রেণিতে সম্পর্কিত লেবেলগুলিতে সবচেয়ে উত্তম ফলাফল দেখিয়েছে, 72% সঠিক বিশ্লেষণ ধরন প্রদর্শিত করে। একইভাবে, LLaMA 3.1 এবং LLaMA 3.2 যথাক্রমে 53% এবং 56% সঠিক ফলাফল প্রদর্শন করেছে। এই গবেষণার মূল্যায়ণ উচ্চকনিষ্ঠ ল্যাঙুয়েজ মডেলগুলির দক্ষতা, বাংলা NLP-এর প্রাসঙ্গিক এবং অপ্রাসঙ্গিক ডেটাসেট থাকার মুখ্য অভাব ছাড়িয়েও। ভবিষ্যতের গবেষণা আরও মডেলগুলির পরীক্ষা, শ্রেণি অসমতাকে দখল করার জন্য, এবং বিশ্লেষণ সহজবোধ্য উপায়গুলির পরিষ্কার ফরমাট করা দিগন্তে আকৃষ্ট হয়।
https://arxiv.org/abs/2601.12132
Large language models (LLMs) are challenging to deploy for domain-specific tasks due to their massive scale. While distilling a fine-tuned LLM into a smaller student model is a promising alternative, the capacity gap between teacher and student often leads to suboptimal performance. This raises a key question: when and how can a student model match or even surpass its teacher on domain-specific tasks? In this work, we propose a novel theoretical insight: a student can outperform its teacher if its advantage on a Student-Favored Subdomain (SFS) outweighs its deficit on the Teacher-Favored Subdomain (TFS). Guided by this insight, we propose Scheduled Checkpoint Distillation (SCD), which reduces the TFS deficit by emulating the teacher's convergence process during supervised fine-tuning (SFT) on the domain task, and a sample-wise Adaptive Weighting (AW) mechanism to preserve student strengths on SFS. Experiments across diverse domain tasks--including QA, NER, and text classification in multiple languages--show that our method consistently outperforms existing distillation approaches, allowing the student model to match or even exceed the performance of its fine-tuned teacher.
大型语言模型(LLMs)由于其巨大的规模,很难针对特定领域任务进行部署。尽管将经过微调的大型语言模型提炼成较小的学生模型是一种有前景的替代方案,但教师和学生模型之间的容量差距通常会导致次优性能。这引发了一个关键问题:在何时何地,学生模型才能匹配甚至超越其教师模型在特定领域的表现?在这项工作中,我们提出了一种新颖的理论见解:如果学生模型在其偏好的子领域(SFS)上的优势能够抵消它在教师偏好子领域(TFS)上的劣势,则该学生模型可以超过其老师的表现。根据这一洞察,我们提出了有计划检查点蒸馏(SCD),这种方法通过模拟教师模型在监督微调(SFT)过程中的收敛情况来减少TFS的差距,并且采用自适应加权机制来保持学生模型在SFS上的优势。实验结果显示,在包括问答、命名实体识别和多语言文本分类等不同领域的任务上,我们的方法始终优于现有的蒸馏方法,使得学生模型能够与甚至超越其微调后的教师模型的表现。
https://arxiv.org/abs/2601.10114