Abstract
Polarity detection becomes substantially more challenging under domain shift, particularly in heterogeneous, long-form narratives with complex discourse structure, such as Holocaust oral histories. This paper presents a corpus-scale diagnostic study of off-the-shelf sentiment classifiers on long-form Holocaust oral histories, using three pretrained transformer-based polarity classifiers on a corpus of 107,305 utterances and 579,013 sentences. After assembling model outputs, we introduce an agreement-based stability taxonomy (ABC) to stratify inter-model output stability. We report pairwise percent agreement, Cohen kappa, Fleiss kappa, and row-normalized confusion matrices to localize systematic disagreement. As an auxiliary descriptive signal, a T5-based emotion classifier is applied to stratified samples from each agreement stratum to compare emotion distributions across strata. The combination of multi-model label triangulation and the ABC taxonomy provides a cautious, operational framework for characterizing where and how sentiment models diverge in sensitive historical narratives. Inter-model agreement is low to moderate overall and is driven primarily by boundary decisions around neutrality.
Abstract (translated)
在领域迁移条件下,极性检测任务变得极具挑战性,尤其在异质性长篇叙事且具有复杂话语结构的场景中,例如大屠杀口述历史。本文对现成情感分类器在长篇大屠杀口述历史语料上的表现进行了大规模诊断研究,使用三个基于预训练转换器的极性分类器,处理包含107,305条话语和579,013个句子的语料库。在整合模型输出后,我们提出了一种基于一致性的稳定性分类法(ABC),以分层呈现模型间输出稳定性。我们报告了配对百分比一致性、科恩卡帕系数、弗利斯卡帕系数以及行归一化混淆矩阵,以定位系统性分歧。作为辅助描述性信号,我们基于T5的情感分类器被应用于各一致性分层的样本,以比较不同层之间的情感分布。多模型标签三角测量与ABC分类法的结合,为刻画情感模型在敏感历史叙事中分歧的方位与方式,提供了谨慎且可操作的框架。整体模型间一致性处于低至中等水平,且主要受限于围绕中立性的边界判定。
URL
https://arxiv.org/abs/2603.28913