Abstract
Financial sentiment analysis (FSA) presents unique challenges to LLMs that surpass those in typical sentiment analysis due to the nuanced language used in financial contexts. The prowess of these models is often undermined by the inherent subjectivity of sentiment classifications in existing benchmark datasets like Financial Phrasebank. These datasets typically feature undefined sentiment classes that reflect the highly individualized perspectives of annotators, leading to significant variability in annotations. This variability results in an unfair expectation for LLMs during benchmarking, where they are tasked to conjecture the subjective viewpoints of human annotators without sufficient context. In this paper, we introduce the Annotators' Instruction Assisted Prompt, a novel evaluation prompt designed to redefine the task definition of FSA for LLMs. By integrating detailed task instructions originally intended for human annotators into the LLMs' prompt framework, AIAP aims to standardize the understanding of sentiment across both human and machine interpretations, providing a fair and context-rich foundation for sentiment analysis. We utilize a new dataset, WSBS, derived from the WallStreetBets subreddit to demonstrate how AIAP significantly enhances LLM performance by aligning machine operations with the refined task definitions. Experimental results demonstrate that AIAP enhances LLM performance significantly, with improvements up to 9.08. This context-aware approach not only yields incremental gains in performance but also introduces an innovative sentiment-indexing method utilizing model confidence scores. This method enhances stock price prediction models and extracts more value from the financial sentiment analysis, underscoring the significance of WSB as a critical source of financial text. Our research offers insights into both improving FSA through better evaluation methods.
Abstract (translated)
金融情感分析（FSA）对大型语言模型（LLM）提出了超出常规情感分析的独特挑战，这是因为金融环境中使用的语言具有复杂性。现有基准数据集如Financial Phrasebank中的情感分类本质上是主观的，这削弱了这些模型的能力。这些数据集中通常包含由标注者个体化视角反映、但未定义的情感类别，导致标注结果存在显著差异。这种变异性在对LLM进行基准测试时造成了不公平的期望，即要求模型推测缺乏足够背景信息的人类标注者的主观观点。 在这篇论文中，我们引入了“注释员指令辅助提示”（AIAP），这是一种重新定义FSA任务的新颖评估提示框架。通过将原本为人类注释者设计的详细任务说明整合到LLM的提示框架中，AIAP旨在标准化人机在情感理解上的共同认知，并提供一个更加公平且背景丰富的基础来进行情感分析。我们利用一个新的数据集WSBS（来自WallStreetBets subreddit的数据）来展示如何通过将机器操作与细化的任务定义对齐，AIAP显著提升了LLM的性能。实验结果表明，AIAP能够大幅度提高LLM的表现，最高可达9.08%的增长。 这种方法不仅带来了性能上的增量改进，还引入了一种新颖的情感指数方法，利用模型置信度分数来增强股票价格预测模型，并从金融情感分析中提取更多价值。这凸显了WSB作为重要金融文本来源的重要性。我们的研究为通过更好的评估方法提升FSA提供了见解。
URL
https://arxiv.org/abs/2505.07871
