Abstract
We investigate the effectiveness of large language models (LLMs), including reasoning-based and non-reasoning models, in performing zero-shot financial sentiment analysis. Using the Financial PhraseBank dataset annotated by domain experts, we evaluate how various LLMs and prompting strategies align with human-labeled sentiment in a financial context. We compare three proprietary LLMs (GPT-4o, GPT-4.1, o3-mini) under different prompting paradigms that simulate System 1 (fast and intuitive) or System 2 (slow and deliberate) thinking and benchmark them against two smaller models (FinBERT-Prosus, FinBERT-Tone) fine-tuned on financial sentiment analysis. Our findings suggest that reasoning, either through prompting or inherent model design, does not improve performance on this task. Surprisingly, the most accurate and human-aligned combination of model and method was GPT-4o without any Chain-of-Thought (CoT) prompting. We further explore how performance is impacted by linguistic complexity and annotation agreement levels, uncovering that reasoning may introduce overthinking, leading to suboptimal predictions. This suggests that for financial sentiment classification, fast, intuitive "System 1"-like thinking aligns more closely with human judgment compared to "System 2"-style slower, deliberative reasoning simulated by reasoning models or CoT prompting. Our results challenge the default assumption that more reasoning always leads to better LLM decisions, particularly in high-stakes financial applications.
Abstract (translated)
我们研究了大型语言模型（LLMs），包括基于推理的和非推理模型，在零样本金融情感分析中的有效性。使用由领域专家标注的Financial PhraseBank数据集，我们评估了各种LLM和提示策略在财务背景下与人类标记的情感的一致性。我们在不同的提示范式下比较了三种专有LLM（GPT-4o、GPT-4.1 和 o3-mini），这些范式模拟了系统1（快速且直觉）或系统2（缓慢且深思熟虑）的思考方式，并将其与两个较小的模型（FinBERT-Prosus和FinBERT-Tone，后者在金融情感分析上进行了微调）进行基准比较。我们的研究结果表明，在这项任务中，无论是通过提示还是内置的设计，推理并不能提高性能。令人惊讶的是，最准确且与人类判断一致的模型方法组合是GPT-4o，没有使用任何链式思维（CoT）提示。 我们进一步探讨了语言复杂性和标注一致性水平对性能的影响，发现推理可能会导致过度思考，从而产生次优预测。这表明，在金融情感分类中，“系统1”式的快速、直觉性思考与人类判断更为契合，而由推理模型或CoT提示模拟的“系统2”式缓慢且深思熟虑的推理则不然。 我们的研究结果挑战了默认假设：更多的推理总是能带来更好的LLM决策，尤其是在高风险金融应用中。
URL
https://arxiv.org/abs/2506.04574
