Deep neural networks have achieved remarkable success across a variety of tasks, yet they often suffer from unreliable probability estimates. As a result, they can be overconfident in their predictions. Conformal Prediction (CP) offers a principled framework for uncertainty quantification, yielding prediction sets with rigorous coverage guarantees. Existing conformal training methods optimize for overall set size, but shaping the prediction sets in a class-conditional manner is not straightforward and typically requires prior knowledge of the data distribution. In this work, we introduce Class Adaptive Conformal Training (CaCT), which formulates conformal training as an augmented Lagrangian optimization problem that adaptively learns to shape prediction sets class-conditionally without making any distributional assumptions. Experiments on multiple benchmark datasets, including standard and long-tailed image recognition as well as text classification, demonstrate that CaCT consistently outperforms prior conformal training methods, producing significantly smaller and more informative prediction sets while maintaining the desired coverage guarantees.
深度神经网络在多种任务中取得了显著的成功,但它们常常遭受不可靠的概率估计问题。因此,这些模型在其预测上可能过于自信。一致性预测(CP)提供了一个基于原则的不确定性量化框架,可以生成具有严格覆盖保证的预测集。现有的符合性训练方法优化的是整体集合大小,但是以条件分类的方式塑造预测集合并不简单,并且通常需要对数据分布有先验知识。 在本研究中,我们引入了类自适应一致性训练(CaCT),该方法将一致性的训练公式化为一个增强拉格朗日优化问题,在不做出任何关于分布假设的情况下,可以学习以条件分类的方式动态调整预测集合。在多个基准数据集上的实验表明,无论是标准和长尾图像识别还是文本分类任务,CaCT都能持续优于先前的一致性训练方法,生成更为精简且信息量更大的预测集合,同时保持预期的覆盖保证。
https://arxiv.org/abs/2601.09522
We consider the problem of distinguishing human-written creative fiction (excerpts from novels) from similar text generated by an LLM. Our results show that, while human observers perform poorly (near chance levels) on this binary classification task, a variety of machine-learning models achieve accuracy in the range 0.93 - 0.98 over a previously unseen test set, even using only short samples and single-token (unigram) features. We therefore employ an inherently interpretable (linear) classifier (with a test accuracy of 0.98), in order to elucidate the underlying reasons for this high accuracy. In our analysis, we identify specific unigram features indicative of LLM-generated text, one of the most important being that the LLM tends to use a larger variety of synonyms, thereby skewing the probability distributions in a manner that is easy to detect for a machine learning classifier, yet very difficult for a human observer. Four additional explanation categories were also identified, namely, temporal drift, Americanisms, foreign language usage, and colloquialisms. As identification of the AI-generated text depends on a constellation of such features, the classification appears robust, and therefore not easy to circumvent by malicious actors intent on misrepresenting AI-generated text as human work.
我们研究了区分人类创作的创意小说(小说片段)和类似由大型语言模型生成文本的问题。我们的结果显示,虽然人类观察者在这项二元分类任务中表现不佳(接近随机水平),但多种机器学习模型在对先前未见过的数据集进行测试时达到了0.93到0.98之间的准确率,即使只使用短样本和单词特征(一元特征)也是如此。因此,我们采用了一种固有的可解释的线性分类器(其测试精度为0.98),以揭示这种高准确性背后的原因。在我们的分析中,我们确定了一些特定的一元特征,这些特征表明了由大型语言模型生成的文本的特点,其中最重要的是:大型语言模型倾向于使用更多种类的同义词,从而使概率分布向易于机器学习分类器检测但对人类观察者来说非常难以察觉的方向偏移。此外还确定了四个额外的解释类别,即时间漂移、美式英语表达、外语用法和俚语。由于识别AI生成文本依赖于多种此类特征的组合,因此这种分类显得十分稳健,并且不容易被试图将AI生成内容冒充为人类作品的恶意行为者所规避。
https://arxiv.org/abs/2601.07368
Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies--element-wise addition, concatenation with projection, and scalar gated fusion--under identical Transformer architectures, data splits, and random seeds. Experiments on three text classification datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than stochastic, we perform paired-seed analysis and cross-dataset comparison across sequence-length regimes. Additional experiments on the ArXiv dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.
翻译如下: 转换器(Transformers)需要位置编码来表示序列顺序,然而大多数先前的工作主要集中在设计新的位置编码上,而不是研究如何将位置信息与标记嵌入融合。在这篇论文中,我们探讨了融合机制本身是否会影响性能,特别是在长序列设置下。我们在相同的Transformer架构、数据分割和随机种子条件下,对三种标准的融合策略——元素级相加、连接后投影以及标量门控融合进行了有控制的经验研究。在三个文本分类数据集(包括短文AG News、中等长度IMDB和长文档ArXiv)上的实验表明,在短文中选择融合方式几乎不会影响性能,但在长文档上则会产生一致的改进效果。为了验证这些改进是结构性而非随机性的,我们进行了配对种子分析及跨序列长度范围的数据集比较。在ArXiv数据集上的额外实验证明了可学习的融合方法能够跨多种位置编码家族进行泛化。最后,我们探索了一种轻量级的卷积门控机制,在长文档上对其进行评估,以引入局部归纳偏差。我们的结果表明,对于长序列转换器而言,位置编码的融合是一个重要的设计选择,并且应该被视为一个明确的建模决策而非固定的默认选项。
https://arxiv.org/abs/2601.05807
This paper proposes an automatic speech recognition (ASR) model for hate speech using large language models (LLMs). The proposed method integrates the encoder of the ASR model with the decoder of the LLMs, enabling simultaneous transcription and censorship tasks to prevent the exposure of harmful content. Instruction tuning of the LLM to mask hate-related words with specific tokens requires an annotated hate speech dataset, which is limited. We generate text samples using an LLM with the Chain-of-Thought (CoT) prompting technique guided by cultural context and examples and then convert them into speech samples using a text-to-speech (TTS) system. However, some of them contain non-hate speech samples with hate-related words, which degrades the censorship performance. This paper filters the samples which text classification models correctly label as hate content. By adjusting the threshold for the number of correct answer models, we can control the level of hate in the generated dataset, allowing us to train the LLMs through curriculum learning in a gradual manner. Experimental results show that the proposed method achieves a masking accuracy of 58.6\% for hate-related words, surpassing previous baselines. We also confirm that the curriculum training contributes to the efficiency of both transcription and censorship tasks.
本文提出了一种使用大型语言模型(LLM)进行仇恨言论自动语音识别(ASR)的模型。该方法将ASR模型的编码器与LLMs的解码器相结合,使得转录和审查任务能够同时执行,以防止有害内容的传播。为了通过特定标记屏蔽与仇恨相关的词汇对LLMs进行指令微调,需要一个带有标注的仇恨言论数据集,但这样的数据集较为有限。我们使用了大型语言模型,并结合了“思维链”(CoT)提示技术,在文化背景和示例引导下生成文本样本,然后通过文字转语音(TTS)系统将其转换为语音样本。然而,部分生成的样本中包含含有仇恨相关词汇但不属于仇恨言论的非仇恨言论样本,这降低了审查性能。本文的方法筛选出被文本分类模型正确标注为仇恨内容的样本。通过调整判定为正确答案的模型数量阈值,我们可以控制生成数据集中的仇恨程度,从而允许我们逐步进行课程学习训练LLMs。实验结果显示,该方法在与仇恨相关的词汇屏蔽准确率上达到了58.6%,超过了之前的基准水平。此外,我们也确认了课程训练对于转录和审查任务的有效性有所提升。
https://arxiv.org/abs/2601.04654
User-Defined Text Classification (UDTC) considers the challenge of classifying input text to user-specified, previously unseen classes, a setting that arises frequently in real-world applications such as enterprise analytics, content moderation, and domain-specific information retrieval. We propose a soft-contextualized encoder architecture for UDTC which contextualizes each candidate label with the label set and a static soft prompt representation of the input query. Training on diverse, multi-source datasets enables the model to generalize effectively to zero-shot classification over entirely unseen topic sets drawn from arbitrary domains. We evaluate the proposed architecture both on held-out in-distribution test data and on multiple unseen UDTC benchmarks. Across datasets, the model achieves state-of-the-art performance, consistently outperforming or matching the baselines.
用户定义文本分类(UDTC)旨在解决将输入文本分类到用户指定且之前未见过的类别中的挑战,这一问题在企业分析、内容审核和特定领域的信息检索等实际应用中经常出现。我们提出了一种软上下文编码器架构来处理 UDTC 任务,该架构通过结合标签集以及输入查询的静态软提示表示来对每个候选标签进行上下文化处理。基于多样化多源数据集的训练使模型能够有效地实现零样本分类,适用于从任意领域抽取的完全未见过的主题集合。我们不仅在保留的内部测试数据上评估了所提出的架构,还在多个未见过的 UDTC 基准测试中进行了测试。跨不同数据集来看,该模型实现了最先进的性能,在所有情况下都优于或与基线方法相匹配。
https://arxiv.org/abs/2601.03450
Semantic text classification has undergone significant advances in recent years due to the rise of large language models (LLMs) and their high dimensional embeddings. While LLM-embeddings are frequently used to store and retrieve text by semantic similarity in vector databases, the global structure semantic relationships in text corpora often remains opaque. Herein we propose a nested density clustering approach, to infer hierarchical trees of semantically related texts. The method starts by identifying texts of strong semantic similarity as it searches for dense clusters in LLM embedding space. As the density criterion is gradually relaxed, these dense clusters merge into more diffuse clusters, until the whole dataset is represented by a single cluster - the root of the tree. By embedding dense clusters into increasingly diffuse ones, we construct a tree structure that captures hierarchical semantic relationships among texts. We outline how this approach can be used to classify textual data for abstracts of scientific abstracts as a case study. This enables the data-driven discovery research areas and their subfields without predefined categories. To evaluate the general applicability of the method, we further apply it to established benchmark datasets such as the 20 News- groups and IMDB 50k Movie Reviews, demonstrating its robustness across domains. Finally we discuss possible applications on scientometrics, topic evolution, highlighting how nested density trees can reveal semantic structure and evolution in textual datasets.
近年来,由于大型语言模型(LLMs)和高维嵌入的兴起,语义文本分类取得了显著进展。尽管LLM-嵌入常用于通过向量数据库存储和检索具有语义相似性的文本,但文本语料库中的全局结构化语义关系往往仍然是不透明的。在此我们提出了一种嵌套密度聚类方法,以推断相关文本的语义层次树。该方法从在LLM嵌入空间中搜索密集簇开始,识别出具有强烈语义相似性的文本。随着密度标准逐渐放松,这些密集簇合并为更稀疏的簇,直到整个数据集由一个单一的簇表示——即树的根部。通过将密集簇嵌入到越来越稀疏的簇中,我们构建了一种结构,该结构捕捉了文本之间的层次语义关系。 本文阐述了如何使用这种方法作为案例研究来对科学摘要的文本来进行分类,这使数据驱动地发现研究领域及其子领域的过程不再需要预定义类别。为了评估该方法的一般适用性,我们将进一步将其应用于20新闻组和IMDb 5万电影评论等已建立基准数据集上,证明了其在不同领域中的鲁棒性。 最后我们讨论了可能的应用场景,包括科学计量学、主题演变等方面,强调嵌套密度树如何揭示文本数据集中的语义结构及其演化。
https://arxiv.org/abs/2512.23471
Text classification plays an important role in various downstream text-related tasks, such as sentiment analysis, fake news detection, and public opinion analysis. Recently, text classification based on Graph Neural Networks (GNNs) has made significant progress due to their strong capabilities of structural relationship learning. However, these approaches still face two major limitations. First, these approaches fail to fully consider the diverse structural information across word pairs, e.g., co-occurrence, syntax, and semantics. Furthermore, they neglect sequence information in the text graph structure information learning module and can not classify texts with new words and relations. In this paper, we propose a Novel Graph-Sequence Learning Model for Inductive Text Classification (TextGSL) to address the previously mentioned issues. More specifically, we construct a single text-level graph for all words in each text and establish different edge types based on the diverse relationships between word pairs. Building upon this, we design an adaptive multi-edge message-passing paradigm to aggregate diverse structural information between word pairs. Additionally, sequential information among text data can be captured by the proposed TextGSL through the incorporation of Transformer layers. Therefore, TextGSL can learn more discriminative text representations. TextGSL has been comprehensively compared with several strong baselines. The experimental results on diverse benchmarking datasets demonstrate that TextGSL outperforms these baselines in terms of accuracy.
文本分类在诸如情感分析、假新闻检测和公众舆论分析等各种下游与文本相关的任务中扮演着重要角色。最近,基于图神经网络(Graph Neural Networks, GNNs)的文本分类方法由于其强大的结构关系学习能力取得了显著进展。然而,这些方法仍然面临着两个主要限制:首先,它们未能充分考虑词汇对之间多样化的结构性信息,如共现、句法和语义;其次,它们忽视了在文本图结构信息学习模块中的序列信息,并且无法分类含有新词和关系的文本。 为了应对上述问题,在本文中我们提出了一种新颖的图-序列学习模型(TextGSL),用于归纳式文本分类。具体来说,我们在每个文本级别上构建了一个包含所有词汇的单一文本级图,并根据词汇对之间的多样化关系建立了不同类型的边。在此基础上,我们设计了自适应多边消息传递范例来聚合词汇对之间多样化的结构信息。此外,通过引入Transformer层,TextGSL可以捕捉到数据中的序列信息。因此,TextGSL能够学习更具区分性的文本表示。 我们在多种强大的基准线上全面比较了TextGSL的表现,并在一系列不同的基准测试数据集上展示了实验结果:结果显示,TextGSL在准确性方面优于这些基线模型。
https://arxiv.org/abs/2512.20097
This tutorial (this https URL) is designed for NLP practitioners, researchers, and developers working with multilingual and low-resource languages who seek to create more equitable and socially impactful language technologies. Participants will walk away with a practical toolkit for building end-to-end NLP pipelines for underrepresented languages -- from data collection and web crawling to parallel sentence mining, machine translation, and downstream applications such as text classification and multimodal reasoning. The tutorial presents strategies for tackling the challenges of data scarcity and cultural variance, offering hands-on methods and modeling frameworks. We will focus on fair, reproducible, and community-informed development approaches, grounded in real-world scenarios. We will showcase a diverse set of use cases covering over 10 languages from different language families and geopolitical contexts, including both digitally resource-rich and severely underrepresented languages.
本教程(此 HTTPS URL)专为从事多语言和低资源语言工作的自然语言处理从业人员、研究人员和开发人员设计,旨在创建更加公平且具有社会影响力的语言技术。参与者将获得一套实用工具包,用于构建面向边缘化语言的端到端自然语言处理管道——从数据收集和网络抓取到平行句子挖掘、机器翻译以及文本分类和多模态推理等下游应用。本教程提供了应对数据稀缺和文化差异挑战的战略,并提供动手操作的方法和建模框架。我们将重点关注公平、可重复且以社区为导向的发展方法,这些方法基于现实世界的情况。我们将展示一系列多样化的用例,涵盖来自不同语言家族和地区背景的10多种语言,包括数字化资源丰富和严重边缘化使用的语言。
https://arxiv.org/abs/2512.14576
We explore efficient strategies to fine-tune decoder-only Large Language Models (LLMs) for downstream text classification under resource constraints. Two approaches are investigated: (1) attaching a classification head to a pre-trained causal LLM and fine-tuning on the task (using the LLM's final token embedding as a sequence representation), and (2) instruction-tuning the LLM in a prompt->response format for classification. To enable single-GPU fine-tuning of models up to 8B parameters, we combine 4-bit model quantization with Low-Rank Adaptation (LoRA) for parameter-efficient training. Experiments on two datasets - a proprietary single-label dataset and the public WIPO-Alpha patent dataset (extreme multi-label classification) - show that the embedding-based method significantly outperforms the instruction-tuned method in F1-score, and is very competitive with - even surpassing - fine-tuned domain-specific models (e.g. BERT) on the same tasks. These results demonstrate that directly leveraging the internal representations of causal LLMs, along with efficient fine-tuning techniques, yields impressive classification performance under limited computational resources. We discuss the advantages of each approach while outlining practical guidelines and future directions for optimizing LLM fine-tuning in classification scenarios.
我们探讨了在资源受限条件下,为下游文本分类任务高效微调解码器独享的大规模语言模型(LLMs)的策略。研究调查了两种方法:(1) 将分类头附加到预训练的因果LMM上,并针对特定任务进行微调(使用LMM最终的标记嵌入作为序列表示),以及 (2) 采用指令调整,以提示-响应格式对LLM进行微调用于分类。为了实现单GPU环境下高达80亿参数模型的有效微调,我们结合了4位模型量化与低秩适应(LoRA)技术,以达到高效的训练效果。 在两个数据集上进行了实验:一个是专有的单一标签数据集和公开的WIPO-Alpha专利数据集(极端多标签分类)。实验结果显示,基于嵌入的方法在F1分数方面显著优于指令调优方法,并且与针对特定领域的微调模型(如BERT)相比,在相同的任务中表现出色甚至超越。 这些结果表明,在计算资源有限的情况下,直接利用因果LMM内部表示以及高效的微调技术可以实现卓越的分类性能。我们讨论了每种方法的优势,并提出了优化LLM在分类场景下的微调的实际指南和未来方向。
https://arxiv.org/abs/2512.12677
Semantic distance measurement is a fundamental problem in computational linguistics, providing a quantitative characterization of similarity or relatedness between text segments, and underpinning tasks such as text retrieval and text classification. From a mathematical perspective, a semantic distance can be viewed as a metric defined on a space of texts or on a representation space derived from them. However, most classical semantic distance methods are essentially fixed, making them difficult to adapt to specific data distributions and task requirements. In this paper, a semantic distance measure based on multi-kernel Gaussian processes (MK-GP) was proposed. The latent semantic function associated with texts was modeled as a Gaussian process, with its covariance function given by a combined kernel combining Matérn and polynomial components. The kernel parameters were learned automatically from data under supervision, rather than being hand-crafted. This semantic distance was instantiated and evaluated in the context of fine-grained sentiment classification with large language models under an in-context learning (ICL) setup. The experimental results demonstrated the effectiveness of the proposed measure.
语义距离测量是计算语言学中的一个基本问题,它提供了一种对文本片段之间相似性或相关性的定量描述,并支持诸如文本检索和分类等任务。从数学角度来看,语义距离可以被视为在文本空间或由其衍生的表示空间上定义的一种度量标准。然而,大多数传统的语义距离方法本质上是固定的,这使得它们难以适应特定的数据分布和任务需求。 本文提出了一种基于多核高斯过程(MK-GP)的语义距离测量方法。该方法将与文本相关的潜在语义函数建模为高斯过程,并使用由Matérn核和多项式成分结合而成的协方差函数来定义其内核参数。这些内核参数是从数据中自动学习得到的,而不是手工设置的。这种语义距离在大规模语言模型的细粒度情感分类任务中的情境学习(ICL)环境中进行了实例化并进行评估。 实验结果表明了所提出的方法的有效性。
https://arxiv.org/abs/2512.12238
LabelFusion is a fusion ensemble for text classification that learns to combine a traditional transformer-based classifier (e.g., RoBERTa) with one or more Large Language Models (LLMs such as OpenAI GPT, Google Gemini, or DeepSeek) to deliver accurate and cost-aware predictions across multi-class and multi-label tasks. The package provides a simple high-level interface (AutoFusionClassifier) that trains the full pipeline end-to-end with minimal configuration, and a flexible API for advanced users. Under the hood, LabelFusion integrates vector signals from both sources by concatenating the ML backbone's embeddings with the LLM-derived per-class scores -- obtained through structured prompt-engineering strategies -- and feeds this joint representation into a compact multi-layer perceptron (FusionMLP) that produces the final prediction. This learned fusion approach captures complementary strengths of LLM reasoning and traditional transformer-based classifiers, yielding robust performance across domains -- achieving 92.4% accuracy on AG News and 92.3% on 10-class Reuters 21578 topic classification -- while enabling practical trade-offs between accuracy, latency, and cost.
LabelFusion 是一种用于文本分类的融合集成方法,它能够学习将传统的基于变压器的分类器(例如 RoBERTa)与一个或多个大型语言模型(LLM,如 OpenAI GPT、Google Gemini 或 DeepSeek)结合在一起,以在多类和多标签任务中提供准确且成本意识强的预测。该软件包提供了简单易用的高级接口 (AutoFusionClassifier),可以进行端到端的完整流水线训练,并需要最少的配置;同时,它还为高级用户提供了一个灵活的 API。 从内部机制来看,LabelFusion 通过将机器学习骨干模型(ML backbone)的嵌入与大型语言模型(LLM)生成的每个类别的得分相连接来整合来自两个来源向量信号——这些得分是通过结构化提示工程策略获得的。然后将这种联合表示输入到一个紧凑的多层感知器(FusionMLP)中,以产生最终预测结果。 这种方法通过学习融合能够捕捉大型语言模型推理和传统基于变压器分类器各自的优势,在多个领域中均表现出稳健性能——在 AG News 数据集上达到了 92.4% 的准确率,在包含十个类别的 Reuters 21578 主题分类数据集上则为 92.3%,同时还能实现准确性、延迟和成本之间的实用权衡。
https://arxiv.org/abs/2512.10793
This study proposes a text classification algorithm based on large language models, aiming to address the limitations of traditional methods in capturing long-range dependencies, understanding contextual semantics, and handling class imbalance. The framework includes text encoding, contextual representation modeling, attention-based enhancement, feature aggregation, and classification prediction. In the representation stage, deep semantic embeddings are obtained through large-scale pretrained language models, and attention mechanisms are applied to enhance the selective representation of key features. In the aggregation stage, global and weighted strategies are combined to generate robust text-level vectors. In the classification stage, a fully connected layer and Softmax output are used to predict class distributions, and cross-entropy loss is employed to optimize model parameters. Comparative experiments introduce multiple baseline models, including recurrent neural networks, graph neural networks, and Transformers, and evaluate them on Precision, Recall, F1-Score, and AUC. Results show that the proposed method outperforms existing models on all metrics, with especially strong improvements in Recall and AUC. In addition, sensitivity experiments are conducted on hyperparameters and data conditions, covering the impact of hidden dimensions on AUC and the impact of class imbalance ratios on Recall. The findings demonstrate that proper model configuration has a significant effect on performance and reveal the adaptability and stability of the model under different conditions. Overall, the proposed text classification method not only achieves effective performance improvement but also verifies its robustness and applicability in complex data environments through systematic analysis.
这项研究提出了一种基于大规模语言模型的文本分类算法,旨在解决传统方法在捕捉长距离依赖性、理解上下文语义以及处理类别不平衡方面的局限。该框架包括文本编码、上下文表示建模、注意力机制增强、特征聚合和分类预测阶段。 在表示阶段,通过大规模预训练的语言模型获得深层语义嵌入,并应用注意机制来增强对关键特征的选择性表示。在聚合阶段,结合全局和加权策略生成稳健的文本级向量。在分类阶段,使用全连接层和Softmax输出来预测类别分布,并采用交叉熵损失来优化模型参数。 比较实验引入了多个基准模型,包括递归神经网络、图神经网络和Transformer,并从精度(Precision)、召回率(Recall)、F1-Score以及AUC等指标评估它们的性能。结果显示,所提出的方法在所有度量标准上均优于现有模型,特别是在召回率和AUC方面有显著改进。 此外,还进行了对超参数和数据条件进行敏感性实验,涵盖了隐藏维度对AUC的影响及类别不平衡比例对召回率的影响。研究发现表明,合适的模型配置会对性能产生重要影响,并揭示了该模型在不同条件下具有适应性和稳定性。 总体而言,所提出的文本分类方法不仅实现了有效的性能提升,还通过系统分析验证了其在复杂数据环境中的鲁棒性和适用性。
https://arxiv.org/abs/2512.09444
Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at this https URL.
前沿语言模型的质量越来越依赖于我们组织网络规模文本语料库的能力进行训练。当前主导的工具在速度和灵活性之间做出权衡:例如,FastText 等词汇分类器速度快但仅限于生成分类输出分数;而变换器文本嵌入模型则可以灵活支持多种工作流程(如聚类、分类及检索),但由于计算成本高昂,并不具备高效性。我们推出了 Luxical 库,该库用于高速“词汇密集”型文本嵌入,旨在结合这两种方法的最佳特性来实现大规模网络文本组织。 Luxical 结合了稀疏的 TF-IDF 特征、一个小的 ReLU 网络和知识蒸馏训练程序,以较低的成本近似大型变换器模型。在这份技术报告中,我们将描述 Luxical 的架构及其培训目标,并在两个不同的应用中评估具体的 Luxical 模型:定向网络爬取文档检索测试及基于文本分类的语言模型数据管理任务。 在这两项任务中,相对于不同大小的神经基线模型,Luxical 能提供从 3 倍到 100 倍的速度提升,并且在数据管理工作流程中接近 FastText 模型推理速度。根据这些评估结果,在大规模文本组织方面,测试过的 Luxical 模型展示出有利的计算/质量权衡,与神经基线模型的质量相当。 Luxical 可以作为开源软件通过此链接访问(请将“this https URL”替换为实际链接地址)。
https://arxiv.org/abs/2512.09015
In this study, we propose a structured methodology that utilizes large language models (LLMs) in a cost-efficient and parsimonious manner, integrating the strengths of scholars and machines while offsetting their respective weaknesses. Our methodology, facilitated through a chain of thought and few-shot learning prompting from computer science, extends best practices for co-author teams in qualitative research to human-machine teams in quantitative research. This allows humans to utilize abductive reasoning and natural language to interrogate not just what the machine has done but also what the human has done. Our method highlights how scholars can manage inherent weaknesses OF LLMs using careful, low-cost techniques. We demonstrate how to use the methodology to interrogate human-machine rating discrepancies for a sample of 1,934 press releases announcing pharmaceutical alliances (1990-2017).
在这项研究中,我们提出了一种结构化的方法论,该方法利用大型语言模型(LLMs),以成本效益高且简洁的方式运作,并整合学者和机器的优势,同时弥补它们各自的不足。我们的方法论通过计算机科学中的思维链和少量样本学习提示来实现,将质性研究中最佳的合作作者团队实践扩展到了量化研究中的人机团队合作。这种方法使得人类能够利用溯因推理和自然语言,不仅质疑机器做了什么,还能质疑人自己做了什么。我们提出的方法强调了学者如何通过仔细且低成本的技术手段来管理LLMs固有的弱点。我们展示了如何使用该方法论来审问一组1,934份宣布医药联盟的新闻稿(1990-2017年)中的人机评分差异。
https://arxiv.org/abs/2512.07583
We present LOCUS (LOw-cost Customization for Universal Specialization), a pipeline that consumes few-shot data to streamline the construction and training of NLP models through targeted retrieval, synthetic data generation, and parameter-efficient tuning. With only a small number of labeled examples, LOCUS discovers pertinent data in a broad repository, synthesizes additional training samples via in-context data generation, and fine-tunes models using either full or low-rank (LoRA) parameter adaptation. Our approach targets named entity recognition (NER) and text classification (TC) benchmarks, consistently outperforming strong baselines (including GPT-4o) while substantially lowering costs and model sizes. Our resultant memory-optimized models retain 99% of fully fine-tuned accuracy while using barely 5% of the memory footprint, also beating GPT-4o on several benchmarks with less than 1% of its parameters.
我们介绍了LOCUS(LOw-cost Customization for Universal Specialization),这是一种通过目标检索、合成数据生成和参数高效调整来消耗少量样本数据,从而简化自然语言处理模型构建和训练的流水线。仅使用少量标记示例,LOCUS可以在广泛的数据存储库中发现相关数据,利用上下文数据生成额外的训练样本,并采用全参数或低秩(LoRA)参数适应方法进行微调。我们的方法针对命名实体识别(NER)和文本分类(TC)基准测试,在性能上持续优于强大的基线模型(包括GPT-4o),同时大幅降低成本和模型规模。我们优化后的内存模型在使用不到全量微调所需内存的5%的情况下,保留了99%的精度,并且在某些基准测试中,参数仅为GPT-4o的不到1%时依然超越了后者的表现。
https://arxiv.org/abs/2512.06239
Large Language Models (LLMs) demonstrate strong in-context learning abilities, yet their effectiveness in text classification depends heavily on prompt design and incurs substantial computational cost. Conformal In-Context Learning (CICLe) has been proposed as a resource-efficient framework that integrates a lightweight base classifier with Conformal Prediction to guide LLM prompting by adaptively reducing the set of candidate classes. However, its broader applicability and efficiency benefits beyond a single domain have not yet been systematically explored. In this paper, we present a comprehensive evaluation of CICLe across diverse NLP classification benchmarks. The results show that CICLe consistently improves over its base classifier and outperforms few-shot prompting baselines when the sample size is sufficient for training the base classifier, and performs comparably in low-data regimes. In terms of efficiency, CICLe reduces the number of shots and prompt length by up to 34.45% and 25.16%, respectively, and enables the use of smaller models with competitive performance. CICLe is furthermore particularly advantageous for text classification tasks with high class imbalance. These findings highlight CICLe as a practical and scalable approach for efficient text classification, combining the robustness of traditional classifiers with the adaptability of LLMs, and achieving substantial gains in data and computational efficiency.
大型语言模型(LLMs)展现了强大的上下文学习能力,但它们在文本分类中的效果严重依赖于提示设计,并且计算成本高。为了解决资源效率问题,提出了符合上下文学习(CICLe)框架,该框架将轻量级的基础分类器与符合预测结合,通过自适应减少候选类别集来指导LLM的提示生成。然而,其在单个领域之外的应用广度和效率优势尚未系统地探索。 本文中,我们对CICLe进行了全面评估,覆盖了多样化的NLP分类基准测试。结果显示,在基础分类器训练所需的样本量充足时,CICLe始终优于它的基础分类器,并且在少量提示基线的条件下表现出色;而在低数据环境中表现相当。 从效率角度来看,CICLe将示例数量和提示长度分别减少了最多34.45%和25.16%,并且允许使用性能竞争的小型模型。此外,在高类别不平衡的文本分类任务中,CICLe尤其具有优势。 这些发现突显了CICLe作为一种实用且可扩展的方法,用于高效的文本分类,结合传统分类器的稳健性与LLM的适应性,并在数据和计算效率方面取得了显著改进。
https://arxiv.org/abs/2512.05732
Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong text classification performance. However, LLM output - here, the category assigned to a text - depends heavily on the wording of the prompt. While literature on prompt engineering is expanding, few studies focus on classification tasks, and even fewer address domains like psychology, where constructs have precise, theory-driven definitions that may not be well represented in pre-training data. We present an empirical framework for optimizing LLM performance for identifying constructs in texts via prompt engineering. We experimentally evaluate five prompting strategies --codebook-guided empirical prompt selection, automatic prompt engineering, persona prompting, chain-of-thought reasoning, and explanatory prompting - with zero-shot and few-shot classification. We find that persona, chain-of-thought, and explanations do not fully address performance loss accompanying a badly worded prompt. Instead, the most influential features of a prompt are the construct definition, task framing, and, to a lesser extent, the examples provided. Across three constructs and two models, the classifications most aligned with expert judgments resulted from a few-shot prompt combining codebook-guided empirical prompt selection with automatic prompt engineering. Based on our findings, we recommend that researchers generate and evaluate as many prompt variants as feasible, whether human-crafted, automatically generated, or ideally both, and select prompts and examples based on empirical performance in a training dataset, validating the final approach in a holdout set. This procedure offers a practical, systematic, and theory-driven method for optimizing LLM prompts in settings where alignment with expert judgment is critical.
由于其架构和庞大的预训练数据,大型语言模型(LLMs)展现了强大的文本分类性能。然而,LLM的输出——即为文本分配的类别——在很大程度上依赖于提示语的措辞。尽管有关提示工程的研究正在扩展,但很少有研究关注分类任务,并且几乎没有研究针对像心理学这样的领域,在这些领域中,构念具有精确、理论驱动的定义,这可能不会很好地反映在预训练数据中。我们提出了一种通过提示工程优化LLM性能以识别文本中构念的经验框架。我们在零样本和少样本分类中实验性地评估了五种提示策略——代码本引导的经验提示选择、自动提示工程、人格提示、链式思维推理以及解释性提示。我们发现,人格、链式思维和解释并不能完全解决因措辞不佳的提示而导致的表现下降问题。相反,提示中最具有影响力的特征是构念定义、任务框架,并且在一定程度上,提供的示例也起到作用。在三个构念和两个模型中进行测试后,最接近专家判断的分类结果来自一种结合了代码本引导的经验提示选择与自动提示工程的少样本提示。基于我们的研究发现,我们建议研究人员尽可能生成并评估各种各样的提示变体,无论是人工设计、自动生成还是理想情况下同时采用这两种方式,并根据训练数据中的实证表现来选择提示和示例,在保留集中进一步验证最终的方法。这一程序为在需要与专家判断保持一致的环境中优化LLM提示提供了一种实用、系统且理论驱动的方法。
https://arxiv.org/abs/2512.03818
Transformer models often exhibit brittle extrapolation, failing on inputs that are longer or structurally more complex than those seen during training. We introduce Counter-Example-Driven Curricula (CEDC), an automated framework that improves model robustness by iteratively focusing on its own failures. At each step, CEDC uses the current model to generate a diverse set of candidate problems, employs a fast, executable verifier to identify incorrect predictions (counter-examples), and then fine-tunes the model on a dataset enriched with these discovered failures. We evaluate CEDC on a suite of algorithmic and natural language tasks, including integer addition, sorting, Dyck-2 language recognition, and three text classification benchmarks. Compared to static training and standard curriculum learning baselines, CEDC achieves up to 30x greater length extrapolation, is 3.75x more computationally efficient than uniform data augmentation, and requires no manual difficulty heuristics. We provide a detailed analysis of the counter-examples, showing how the curriculum naturally adapts to target progressively more complex error modes. Our findings establish verifier-guided, failure-driven learning as a simple, powerful, and efficient paradigm for enhancing the generalization capabilities of Transformer models.
翻译: Transformer 模型常常在面对超出训练数据长度或结构复杂度的输入时表现出脆弱性。我们引入了一种名为“反例驱动课程学习”(CEDC)的自动化框架,通过迭代关注模型自身的错误来提高其鲁棒性。在每一步中,CEDC 使用当前模型生成一系列多样化的候选问题集,使用快速可执行验证器识别出不正确的预测(反例),然后对包含这些发现失败案例的数据集进行微调。 我们在整数加法、排序、Dyck-2 语言识别和三项文本分类基准上评估了 CEDC。与静态训练以及标准课程学习基线相比,CEDC 在长度外推方面取得了高达30倍的进步,其计算效率比均匀数据增强高出3.75倍,并且不需要手动难度启发式方法。 我们提供了对反例的详细分析,展示了该课程如何自然地适应目标并逐渐解决更复杂的错误模式。我们的研究结果确立了由验证器引导、以失败为导向的学习作为一种简单、强大而高效的范式,用于增强 Transformer 模型的一般化能力。
https://arxiv.org/abs/2512.01187
This study presents an unsupervised method to infer discreteness, syntax and temporal structures of fruit-bats vocalizations, as a case study of graded vocal systems, and evaluates the complexity of communication patterns in relation with behavioral context. The method improved the baseline for unsupervised labeling of vocal units (i.e. syllables) through manifold learning, by investigating how dimen- sionality reduction on mel-spectrograms affects labeling, and comparing it with unsupervised labels based on acoustic similarity. We then encoded vocalizations as syllabic sequences to analyze the type of syntax, and extracted the Maximal Repetitions (MRs) to evaluate syntactical structures. We found evidence for: i) associative syntax, rather than combinatorial (context classification is unaffected by permutation of sequences, F 1 > 0.9); ii) context-dependent use of syllables (Wilcoxon rank-sum tests, p-value < 0.05); iii) heavy-tail distribution of MRs (truncated power-law, exponent {\alpha} < 2), indicative of mechanism encoding com- binatorial complexity. Analysis of MRs and syllabic transition networks revealed that mother-pupil interactions were characterized by repetitions, while commu- nication in conflict-contexts exhibited higher complexity (longer MRs and more interconnected vocal sequences) than non-agonistic contexts. We propose that communicative complexity is higher in scenarios of disagreement, reflecting lower compressibility of information.
这项研究提出了一种无监督方法,用于推断果蝠叫声中的离散性、语法结构和时间结构,并以此作为分级声音系统的案例研究。该方法评估了在行为背景下的交流模式的复杂性。 具体而言,本方法通过流形学习改进了基于无监督标签的声音单元(即音节)的基本标注水平,探讨了梅尔频谱图降维对标注的影响,并将其与基于声学相似性的无监督标签进行了比较。然后我们将声音编码为音节序列以分析语法类型,并提取最大重复次数(MRs)来评估语法结构。 研究发现有以下几点证据: i) 关联性语法规则,而不是组合性规则(序列的排列对上下文分类没有影响,F1 > 0.9) ii) 音节使用的背景依赖性(威尔科克森秩和检验,p值 < 0.05) iii) 最大重复次数的幂律分布(截断幂律,指数{\alpha} < 2),表明存在编码组合复杂性的机制 通过对最大重复次数和音节转换网络进行分析发现,母蝙蝠与幼崽之间的互动特征是重复性更强,而冲突背景下的交流则表现出更高的复杂性(更长的MRs以及更加相互连接的声音序列)。研究提出,在分歧情景下,沟通的复杂度更高,反映了信息可压缩性的降低。
https://arxiv.org/abs/2512.01033
Accurately dating historical texts is essential for organizing and interpreting cultural heritage collections. This article addresses temporal text classification using interpretable, feature-engineered tree-based machine learning models. We integrate five feature categories - compression-based, lexical structure, readability, neologism detection, and distance features - to predict the temporal origin of English texts spanning five centuries. Comparative analysis shows that these feature domains provide complementary temporal signals, with combined models outperforming any individual feature set. On a large-scale corpus, we achieve 76.7% accuracy for century-scale prediction and 26.1% for decade-scale classification, substantially above random baselines (20% and 2.3%). Under relaxed temporal precision, performance increases to 96.0% top-2 accuracy for centuries and 85.8% top-10 accuracy for decades. The final model exhibits strong ranking capabilities with AUCROC up to 94.8% and AUPRC up to 83.3%, and maintains controlled errors with mean absolute deviations of 27 years and 30 years, respectively. For authentication-style tasks, binary models around key thresholds (e.g., 1850-1900) reach 85-98% accuracy. Feature importance analysis identifies distance features and lexical structure as most informative, with compression-based features providing complementary signals. SHAP explainability reveals systematic linguistic evolution patterns, with the 19th century emerging as a pivot point across feature domains. Cross-dataset evaluation on Project Gutenberg highlights domain adaptation challenges, with accuracy dropping by 26.4 percentage points, yet the computational efficiency and interpretability of tree-based models still offer a scalable, explainable alternative to neural architectures.
https://arxiv.org/abs/2511.23056