Abstract
Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts.
Abstract (translated)
大型语言模型（LLM）中编码的政治偏见可能对下游应用产生负面影响。现有的偏见分析方法依赖于小型中间任务（如问卷回答或政治内容生成），并且依赖于LLMs本身进行分析，从而传播了偏见。我们提出了一种新方法，利用观察到的LLM情感预测在句子中的目标实体变化这一现象，并定义了一个基于熵的不一致性指标来编码这种预测变动性。 为了测试这种方法，我们在450条政治语句中插入了1319个具有不同人口统计学和政治背景的政治家名字，并使用六种广泛使用的语言中的七种模型进行了目标导向的情感预测。我们观察到所有测试组合中存在不一致性，并在不同的粒度级别上进行了一致的统计分析。 我们发现对左翼和极右翼政客有积极和消极的偏见，且具有相似政治立场的政治家之间存在正相关性。西方语言中的偏见强度高于其他语言。较大的模型表现出更强且更一致的偏见，并减少了类似语言之间的差异。 为了部分缓解LLM在目标导向情感分类（TSC）中的不可靠性，我们用虚构但可信的替身替代了政治家的名字。
URL
https://arxiv.org/abs/2505.19776
