Sentiment_Classification
-
TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu
2025-08-02 20:42:37Vallabhaneni Raj Kumar, Ashwin S, Supriya Manna, Niladri Sett, Cheedella V S N M S Hema Harshitha, Kurakula Harshitha, Anand Kumar Sharma, Basina Deepakraj, Tanuj Sarkar, Bondada Navaneeth Krishna, Samanthapudi ShakeerAbstract
In the Indian subcontinent, Telugu, one of India's six classical languages, is the most widely spoken Dravidian Language. Despite its 96 million speaker base worldwide, Telugu remains underrepresented in the global NLP and Machine Learning landscape, mainly due to lack of high-quality annotated resources. This work introduces TeSent, a comprehensive benchmark dataset for sentiment classification, a key text classification problem, in Telugu. TeSent not only provides ground truth labels for the sentences, but also supplements with provisions for evaluating explainability and fairness, two critical requirements in modern-day machine learning tasks. We scraped Telugu texts covering multiple domains from various social media platforms, news websites and web-blogs to preprocess and generate 26,150 sentences, and developed a custom-built annotation platform and a carefully crafted annotation protocol for collecting the ground truth labels along with their human-annotated rationales. We then fine-tuned several SOTA pre-trained models in two ways: with rationales, and without rationales. Further, we provide a detailed plausibility and faithfulness evaluation suite, which exploits the rationales, for six widely used post-hoc explainers applied on the trained models. Lastly, we curate TeEEC, Equity Evaluation Corpus in Telugu, a corpus to evaluate fairness of Telugu sentiment and emotion related NLP tasks, and provide a fairness evaluation suite for the trained classifier models. Our experimental results suggest that training with rationales may improve model accuracy, reduce bias in models, and make the explainers' output more aligned to human reasoning.
Abstract (translated)
在印度次大陆，泰卢固语（Telugu）是印度六种古典语言之一，也是使用最广泛的德拉威语系语言。尽管其全球使用者数量达到9600万，但由于高质量标注资源的缺乏，泰卢固语在全球自然语言处理和机器学习领域中仍处于相对边缘化的地位。为此，本研究推出了TeSent，一个全面的基准数据集，用于泰卢固语情感分类这一关键文本分类问题的研究。 TeSent不仅提供了句子的真实标签，还补充了评估可解释性和公平性的机制，这两者是现代机器学习任务中的重要需求。我们从各种社交媒体平台、新闻网站和网络博客中抓取了大量的泰卢固语文本，涵盖多个领域，并预处理生成了26,150个句子。同时，我们建立了一个定制的标注平台以及一个精心设计的标注协议来收集这些真实的标签及其人工标注的理由。 接着，我们在两个方面对几个最先进的预训练模型进行了微调：一是带有理由的情况；二是不带理由的情况。此外，我们还提供了一套详细的合理性与忠实性评估工具，这套工具利用了上述理由，并应用于六种广泛使用的后处理解释器，在训练后的模型上进行应用。 最后，为了评估泰卢固语情感和情绪相关的自然语言处理任务的公平性，我们整理出了TeEEC（Telugu Equity Evaluation Corpus），这是一份专门用于评估此类任务中模型公平性的语料库，并且为这些经过训练的分类器提供了一套公平性评估工具。 实验结果表明，使用理由进行训练可以提高模型的准确性，减少模型中的偏见，并使解释器的输出更加符合人类推理。
URL
https://arxiv.org/abs/2508.01486
PDF
-
Understanding Public Perception of Crime in Bangladesh: A Transformer-Based Approach with Explainability
2025-07-28 18:00:25Fatema Binte Hassan, Md Al Jubair, Mohammad Mehadi Hasan, Tahmid Hossain, S M Mehebubur Rahman Khan Shuvo, Mohammad Shamsul ArefinAbstract
In recent years, social media platforms have become prominent spaces for individuals to express their opinions on ongoing events, including criminal incidents. As a result, public sentiment can shift dynamically over time. This study investigates the evolving public perception of crime-related news by classifying user-generated comments into three categories: positive, negative, and neutral. A newly curated dataset comprising 28,528 Bangla-language social media comments was developed for this purpose. We propose a transformer-based model utilizing the XLM-RoBERTa Base architecture, which achieves a classification accuracy of 97%, outperforming existing state-of-the-art methods in Bangla sentiment analysis. To enhance model interpretability, explainable AI technique is employed to identify the most influential features driving sentiment classification. The results underscore the effectiveness of transformer-based models in processing low-resource languages such as Bengali and demonstrate their potential to extract actionable insights that can support public policy formulation and crime prevention strategies.
Abstract (translated)
近年来，社交媒体平台已成为个人表达对当前事件（包括犯罪事件）看法的重要空间。因此，公众情绪会随着时间的推移而动态变化。本研究通过将用户生成的评论分类为三类：积极、消极和中立，来探讨与犯罪相关的新闻报道中的公众认知演变。为此目的开发了一个新整理的数据集，包含28,528条孟加拉语社交媒体评论。 我们提出了一种基于变压器架构（使用XLM-RoBERTa Base）的模型，在孟加拉语情感分析中实现了97%的分类准确率，超越了现有最先进的方法。为了提高模型可解释性，采用了可解释的人工智能技术来识别驱动情绪分类的关键特征。研究结果强调了基于Transformer的模型在处理包括孟加拉语在内的低资源语言中的有效性，并展示了它们提取可用于支持公共政策制定和犯罪预防策略的实用信息的能力。
URL
https://arxiv.org/abs/2507.21234
PDF
-
PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
2025-07-26 21:46:32Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas ShakkottaiAbstract
Inference-time alignment enables large language models (LLMs) to generate outputs aligned with end-user preferences without further training. Recent post-training methods achieve this by using small guidance models to modify token generation during inference. These methods typically optimize a reward function KL-regularized by the original LLM taken as the reference policy. A critical limitation, however, is their dependence on a pre-trained reward model, which requires fitting to human preference feedback--a potentially unstable process. In contrast, we introduce PITA, a novel framework that integrates preference feedback directly into the LLM's token generation, eliminating the need for a reward model. PITA learns a small preference-based guidance policy to modify token probabilities at inference time without LLM fine-tuning, reducing computational cost and bypassing the pre-trained reward model dependency. The problem is framed as identifying an underlying preference distribution, solved through stochastic search and iterative refinement of the preference-based guidance model. We evaluate PITA across diverse tasks, including mathematical reasoning and sentiment classification, demonstrating its effectiveness in aligning LLM outputs with user preferences.
Abstract (translated)
推理时间对齐使大型语言模型（LLM）能够在无需进一步训练的情况下生成与最终用户偏好一致的输出。最近的后期训练方法通过使用小规模引导模型来修改推断过程中的令牌生成，从而实现了这一点。这些方法通常优化一个由原始LLM作为参考策略进行KL正则化的奖励函数。然而，一个关键限制是它们依赖于预训练的奖励模型，该模型需要适应人类偏好的反馈——这是一个可能不稳定的过程。相比之下，我们引入了PITA这一新颖框架，它直接将偏好反馈整合到LLM的令牌生成过程中，从而消除了对奖励模型的需求。PITA学习一个小规模基于偏好的引导策略，在推理时间修改令牌概率而无需对LLM进行微调，这样可以减少计算成本并绕过预训练奖励模型的依赖性。该问题被构架为识别潜在偏好分布的问题，并通过随机搜索和迭代改进基于偏好的引导模型来解决。我们在包括数学推理和情感分类在内的各种任务上评估了PITA，证明了它在将LLM输出与用户偏好对齐方面的有效性。
URL
https://arxiv.org/abs/2507.20067
PDF
-
FinMarBa: A Market-Informed Dataset for Financial Sentiment Classification
2025-07-24 16:27:32Baptiste Lefort, Eric Benhamou, Beatrice Guez, Jean-Jacques Ohana, Ethan Setrouk, Alban EtienneAbstract
This paper presents a novel hierarchical framework for portfolio optimization, integrating lightweight Large Language Models (LLMs) with Deep Reinforcement Learning (DRL) to combine sentiment signals from financial news with traditional market indicators. Our three-tier architecture employs base RL agents to process hybrid data, meta-agents to aggregate their decisions, and a super-agent to merge decisions based on market data and sentiment analysis. Evaluated on data from 2018 to 2024, after training on 2000-2017, the framework achieves a 26% annualized return and a Sharpe ratio of 1.2, outperforming equal-weighted and S&P 500 benchmarks. Key contributions include scalable cross-modal integration, a hierarchical RL structure for enhanced stability, and open-source reproducibility.
Abstract (translated)
本文提出了一种新颖的分层框架，用于投资组合优化，该框架将轻量级大型语言模型（LLMs）与深度强化学习（DRL）相结合，以融合来自金融新闻的情绪信号和传统的市场指标。我们的三层架构包括基础RL代理来处理混合数据、元代理来汇总他们的决策以及超级代理来基于市场数据和情绪分析进行合并决策。在2018年至2024年的数据上进行了评估，在2000年至2017年期间完成了训练，该框架实现了每年化收益率26%及夏普比率1.2的业绩，优于等权重基准和标普500指数。主要贡献包括可扩展的跨模态整合、增强稳定性的分层RL结构以及开源可复制性。
URL
https://arxiv.org/abs/2507.22932
PDF
-
FinDPO: Financial Sentiment Analysis for Algorithmic Trading through Preference Optimization of LLMs
2025-07-24 13:57:05Giorgos Iacovides, Wuyang Zhou, Danilo MandicAbstract
Opinions expressed in online finance-related textual data are having an increasingly profound impact on trading decisions and market movements. This trend highlights the vital role of sentiment analysis as a tool for quantifying the nature and strength of such opinions. With the rapid development of Generative AI (GenAI), supervised fine-tuned (SFT) large language models (LLMs) have become the de facto standard for financial sentiment analysis. However, the SFT paradigm can lead to memorization of the training data and often fails to generalize to unseen samples. This is a critical limitation in financial domains, where models must adapt to previously unobserved events and the nuanced, domain-specific language of finance. To this end, we introduce FinDPO, the first finance-specific LLM framework based on post-training human preference alignment via Direct Preference Optimization (DPO). The proposed FinDPO achieves state-of-the-art performance on standard sentiment classification benchmarks, outperforming existing supervised fine-tuned models by 11% on the average. Uniquely, the FinDPO framework enables the integration of a fine-tuned causal LLM into realistic portfolio strategies through a novel 'logit-to-score' conversion, which transforms discrete sentiment predictions into continuous, rankable sentiment scores (probabilities). In this way, simulations demonstrate that FinDPO is the first sentiment-based approach to maintain substantial positive returns of 67% annually and strong risk-adjusted performance, as indicated by a Sharpe ratio of 2.0, even under realistic transaction costs of 5 basis points (bps).
Abstract (translated)
在线金融相关文本数据中表达的意见对交易决策和市场走势的影响日益显著。这种趋势强调了情感分析作为量化此类意见性质和强度的工具的重要性。随着生成式人工智能（GenAI）的迅速发展，监督微调（SFT）大型语言模型（LLMs）已成为金融市场情感分析的标准方法。然而，SFT范式可能导致对训练数据的记忆化，并且通常无法泛化到未见过的数据样本上。在金融领域中，这种限制尤为关键，因为模型必须适应前所未有的事件并处理金融领域的微妙、特定于行业的语言表达方式。 为此，我们推出了FinDPO——首个基于直接偏好优化（DPO）进行后训练人类偏好数值对齐的金融专用LLM框架。所提出的FinDPO在标准的情感分类基准测试中取得了最先进的性能，在平均精度上超过了现有的监督微调模型11%。独特的是，FinDPO框架通过一种新颖的“logit到评分”转换方法，使经过微调的因果LLM能够整合进实际的投资组合策略中，这种转换将离散的情感预测转化为连续、可排名的情感得分（概率）。这种方式使得模拟结果表明，FinDPO是首个能在每年保持显著正回报率（67%）和强大的风险调整后表现的情况下，采用情感基础方法，并且即使在现实的交易成本为5个基点时也能做到这一点，其夏普比率为2.0。
URL
https://arxiv.org/abs/2507.18417
PDF
-
AdaptiSent: Context-Aware Adaptive Attention for Multimodal Aspect-Based Sentiment Analysis
2025-07-17 00:06:43S M Rafiuddin, Sadia Kamal, Mohammed Rakib, Arunkumar Bagavathi, Atriya SenAbstract
We introduce AdaptiSent, a new framework for Multimodal Aspect-Based Sentiment Analysis (MABSA) that uses adaptive cross-modal attention mechanisms to improve sentiment classification and aspect term extraction from both text and images. Our model integrates dynamic modality weighting and context-adaptive attention, enhancing the extraction of sentiment and aspect-related information by focusing on how textual cues and visual context interact. We tested our approach against several baselines, including traditional text-based models and other multimodal methods. Results from standard Twitter datasets show that AdaptiSent surpasses existing models in precision, recall, and F1 score, and is particularly effective in identifying nuanced inter-modal relationships that are crucial for accurate sentiment and aspect term extraction. This effectiveness comes from the model's ability to adjust its focus dynamically based on the context's relevance, improving the depth and accuracy of sentiment analysis across various multimodal data sets. AdaptiSent sets a new standard for MABSA, significantly outperforming current methods, especially in understanding complex multimodal information.
Abstract (translated)
我们介绍了AdaptiSent，这是一个新的多模态基于方面的情感分析（MABSA）框架。该框架采用自适应跨模式注意力机制来提高从文本和图像中提取情感分类和方面术语的效果。我们的模型整合了动态模态加权和上下文自适应注意功能，通过关注文字线索与视觉背景的相互作用，增强了情感和相关方面的信息抽取。 我们把这种方法与其他基线方法进行了对比测试，包括传统的基于文本的方法和其他多模态方法。在标准Twitter数据集上的结果显示，AdaptiSent在精度、召回率和F1分数上均超越了现有的模型，并且特别擅长识别对于准确的情感和方面术语提取至关重要的细微的跨模式关系。 这种有效性来源于该模型根据上下文的相关性动态调整其关注焦点的能力，从而提高了各种多模态数据集中的情感分析的深度和准确性。AdaptiSent为MABSA设定了新的标准，在理解复杂多模态信息方面显著优于现有的方法。
URL
https://arxiv.org/abs/2507.12695
PDF
-
Large Language Models' Varying Accuracy in Recognizing Risk-Promoting and Health-Supporting Sentiments in Public Health Discourse: The Cases of HPV Vaccination and Heated Tobacco Products
2025-07-06 11:57:02Soojong Kim, Kwanho Kim, Hye Min KimAbstract
Machine learning methods are increasingly applied to analyze health-related public discourse based on large-scale data, but questions remain regarding their ability to accurately detect different types of health sentiments. Especially, Large Language Models (LLMs) have gained attention as a powerful technology, yet their accuracy and feasibility in capturing different opinions and perspectives on health issues are largely unexplored. Thus, this research examines how accurate the three prominent LLMs (GPT, Gemini, and LLAMA) are in detecting risk-promoting versus health-supporting sentiments across two critical public health topics: Human Papillomavirus (HPV) vaccination and heated tobacco products (HTPs). Drawing on data from Facebook and Twitter, we curated multiple sets of messages supporting or opposing recommended health behaviors, supplemented with human annotations as the gold standard for sentiment classification. The findings indicate that all three LLMs generally demonstrate substantial accuracy in classifying risk-promoting and health-supporting sentiments, although notable discrepancies emerge by platform, health issue, and model type. Specifically, models often show higher accuracy for risk-promoting sentiment on Facebook, whereas health-supporting messages on Twitter are more accurately detected. An additional analysis also shows the challenges LLMs face in reliably detecting neutral messages. These results highlight the importance of carefully selecting and validating language models for public health analyses, particularly given potential biases in training data that may lead LLMs to overestimate or underestimate the prevalence of certain perspectives.
Abstract (translated)
基于大规模数据的机器学习方法越来越多地被应用于分析与健康相关的公众言论，但关于它们准确检测不同类型健康情感的能力仍然存在疑问。特别是，大型语言模型（LLMs）作为一项强大的技术引起了关注，然而它们在捕捉健康问题的不同观点和视角方面的准确性及可行性尚未得到充分探索。因此，这项研究考察了三种知名的大规模语言模型（GPT、Gemini 和 LLAMA），它们在检测与两种关键公共卫生话题相关的情感方面——人类乳头瘤病毒（HPV）疫苗接种和加热烟草产品（HTPs）——的准确度如何。我们从 Facebook 和 Twitter 中提取了多组支持或反对推荐健康行为的信息，并通过人工注释作为情感分类的标准来补充这些数据。研究结果表明，这三种 LLM 通常在识别风险促进与健康支持性情感方面表现出相当高的准确性，尽管这种准确性因平台、健康问题和模型类型而异。具体而言，模型往往在 Facebook 上对风险促进的情感检测更准确，而在 Twitter 上则更擅长于检测健康支持性的信息。此外的分析还显示了 LLM 在可靠地检测中立消息时面临的挑战。这些结果突显了在进行公共卫生分析时选择和验证语言模型的重要性，尤其是在考虑到训练数据中存在的潜在偏见可能导致 LLM 对某些观点的流行程度估计过高或过低的情况下。
URL
https://arxiv.org/abs/2507.04364
PDF
-
NEU-ESC: A Comprehensive Vietnamese dataset for Educational Sentiment analysis and topic Classification toward multitask learning
2025-06-30 05:19:04Phan Quoc Hung Mai, Quang Hung Nguyen, Phuong Giang Duong, Hong Hanh Nguyen, Nguyen Tuan LongAbstract
In the field of education, understanding students' opinions through their comments is crucial, especially in the Vietnamese language, where resources remain limited. Existing educational datasets often lack domain relevance and student slang. To address these gaps, we introduce NEU-ESC, a new Vietnamese dataset for Educational Sentiment Classification and Topic Classification, curated from university forums, which offers more samples, richer class diversity, longer texts, and broader vocabulary. In addition, we explore multitask learning using encoder-only language models (BERT), in which we showed that it achieves performance up to 83.7% and 79.8% accuracy for sentiment and topic classification tasks. We also benchmark our dataset and model with other datasets and models, including Large Language Models, and discuss these benchmarks. The dataset is publicly available at: this https URL.
Abstract (translated)
在教育领域，通过学生的评论了解他们的意见至关重要，特别是在越南语中，由于资源有限，这一挑战更加突出。现有的教育数据集通常缺乏领域的相关性和学生用语的多样性。为了弥补这些不足，我们推出了NEU-ESC，这是一个新的越南语数据集，用于教育情感分类和话题分类，该数据集是从大学论坛上收集整理而来，提供了更多的样本、更丰富的类别多样性、更长的文章长度以及更为广泛的词汇量。此外，我们还探索了使用编码器单独的语言模型（如BERT）进行多任务学习，在这种情况下，我们展示了它在情感分类和话题分类任务中分别达到了高达83.7%和79.8%的准确性。我们还用其他数据集和模型包括大型语言模型对我们的数据集和模型进行了基准测试，并讨论了这些基准的结果。该数据集可在此URL公开获取：this https URL。
URL
https://arxiv.org/abs/2506.23524
PDF
-
From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling
2025-06-19 15:26:08Yao Lu, Zhaiyuan Ji, Jiawei Du, Yu Shanqing, Qi Xuan, Tianyi ZhouAbstract
Although the annotation paradigm based on Large Language Models (LLMs) has made significant breakthroughs in recent years, its actual deployment still has two core bottlenecks: first, the cost of calling commercial APIs in large-scale annotation is very expensive; second, in scenarios that require fine-grained semantic understanding, such as sentiment classification and toxicity classification, the annotation accuracy of LLMs is even lower than that of Small Language Models (SLMs) dedicated to this field. To address these problems, we propose a new paradigm of multi-model cooperative annotation and design a fully automatic annotation framework AutoAnnotator based on this. Specifically, AutoAnnotator consists of two layers. The upper-level meta-controller layer uses the generation and reasoning capabilities of LLMs to select SLMs for annotation, automatically generate annotation code and verify difficult samples; the lower-level task-specialist layer consists of multiple SLMs that perform annotation through multi-model voting. In addition, we use the difficult samples obtained by the secondary review of the meta-controller layer as the reinforcement learning set and fine-tune the SLMs in stages through a continual learning strategy, thereby improving the generalization of SLMs. Extensive experiments show that AutoAnnotator outperforms existing open-source/API LLMs in zero-shot, one-shot, CoT, and majority voting settings. Notably, AutoAnnotator reduces the annotation cost by 74.15% compared to directly annotating with GPT-3.5-turbo, while still improving the accuracy by 6.21%. Project page: this https URL.
Abstract (translated)
尽管基于大型语言模型（LLM）的注释范式近年来取得了重大突破，但在实际部署中仍然存在两个核心瓶颈：首先，在大规模标注场景下调用商业API的成本非常高昂；其次，在需要细粒度语义理解的情境下，例如情感分类和毒性分类等任务，LLM的标注准确率甚至低于专门为此领域设计的小型语言模型（SLMs）。为了解决这些问题，我们提出了一种多模态协同注释的新范式，并基于此设计了一个全自动注释框架AutoAnnotator。 具体来说，AutoAnnotator包含两个层次。顶层元控制器层利用LLM的生成和推理能力来选择合适的SLM进行标注，自动生成标注代码并验证难例；底层任务专家层则由多个专精于特定任务的SLM组成，通过多模型投票的方式执行注释工作。此外，我们还采用了元控制器层二次审查得到的难例作为强化学习的数据集，并通过持续学习策略分阶段微调SLMs，从而提高其泛化能力。 广泛的实验表明，在零样本、单样本、CoT（链式推理）和多数表决等设置下，AutoAnnotator的表现优于现有的开源/API LLM。特别值得注意的是，相较于直接使用GPT-3.5-turbo进行标注，AutoAnnotator将注释成本降低了74.15%，同时仍然提高了6.21%的准确性。 项目页面: [请参阅此链接](https://this.is.the.url/project-page/)（请注意替换实际URL）。
URL
https://arxiv.org/abs/2506.16393
PDF
-
Quantum Graph Transformer for NLP Sentiment Classification
2025-06-09 16:55:41Shamminuj Aktar, Andreas B\"artschi, Abdel-Hameed A. Badawy, Stephan EidenbenzAbstract
Quantum machine learning is a promising direction for building more efficient and expressive models, particularly in domains where understanding complex, structured data is critical. We present the Quantum Graph Transformer (QGT), a hybrid graph-based architecture that integrates a quantum self-attention mechanism into the message-passing framework for structured language modeling. The attention mechanism is implemented using parameterized quantum circuits (PQCs), which enable the model to capture rich contextual relationships while significantly reducing the number of trainable parameters compared to classical attention mechanisms. We evaluate QGT on five sentiment classification benchmarks. Experimental results show that QGT consistently achieves higher or comparable accuracy than existing quantum natural language processing (QNLP) models, including both attention-based and non-attention-based approaches. When compared with an equivalent classical graph transformer, QGT yields an average accuracy improvement of 5.42% on real-world datasets and 4.76% on synthetic datasets. Additionally, QGT demonstrates improved sample efficiency, requiring nearly 50% fewer labeled samples to reach comparable performance on the Yelp dataset. These results highlight the potential of graph-based QNLP techniques for advancing efficient and scalable language understanding.
Abstract (translated)
量子机器学习是构建更高效和表达力更强的模型的一个有前景的方向，尤其是在理解复杂、结构化数据至关重要的领域。我们提出了一种基于图的混合架构——量子图变换器（Quantum Graph Transformer, QGT），它在结构化的语言建模中将量子自注意力机制整合到了消息传递框架内。该注意机制通过参数化量子电路（Parameterized Quantum Circuits，PQCs）实现，这使得模型能够捕捉丰富的上下文关系，并且相比传统的经典注意机制显著减少了可训练的参数数量。 我们在五个情感分类基准测试集上评估了QGT的效果。实验结果显示，QGT在准确度方面始终优于现有的量子自然语言处理（Quantum Natural Language Processing, QNLP）模型，包括基于注意力和非注意力的方法。与等效的经典图变换器相比，在真实世界数据集上，QGT的平均精度提高了5.42%，而在合成数据集上的提高为4.76%。此外，QGT还展示了更好的样本效率，仅需大约少一半数量的标记样本即可在Yelp数据集上达到类似性能。 这些结果突显了基于图的QNLP技术在推进高效和可扩展的语言理解方面的潜力。
URL
https://arxiv.org/abs/2506.07937
PDF
-
Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis
2025-06-05 02:47:23Dimitris Vamvourellis, Dhagash MehtaAbstract
We investigate the effectiveness of large language models (LLMs), including reasoning-based and non-reasoning models, in performing zero-shot financial sentiment analysis. Using the Financial PhraseBank dataset annotated by domain experts, we evaluate how various LLMs and prompting strategies align with human-labeled sentiment in a financial context. We compare three proprietary LLMs (GPT-4o, GPT-4.1, o3-mini) under different prompting paradigms that simulate System 1 (fast and intuitive) or System 2 (slow and deliberate) thinking and benchmark them against two smaller models (FinBERT-Prosus, FinBERT-Tone) fine-tuned on financial sentiment analysis. Our findings suggest that reasoning, either through prompting or inherent model design, does not improve performance on this task. Surprisingly, the most accurate and human-aligned combination of model and method was GPT-4o without any Chain-of-Thought (CoT) prompting. We further explore how performance is impacted by linguistic complexity and annotation agreement levels, uncovering that reasoning may introduce overthinking, leading to suboptimal predictions. This suggests that for financial sentiment classification, fast, intuitive "System 1"-like thinking aligns more closely with human judgment compared to "System 2"-style slower, deliberative reasoning simulated by reasoning models or CoT prompting. Our results challenge the default assumption that more reasoning always leads to better LLM decisions, particularly in high-stakes financial applications.
Abstract (translated)
我们研究了大型语言模型（LLMs），包括基于推理的和非推理模型，在零样本金融情感分析中的有效性。使用由领域专家标注的Financial PhraseBank数据集，我们评估了各种LLM和提示策略在财务背景下与人类标记的情感的一致性。我们在不同的提示范式下比较了三种专有LLM（GPT-4o、GPT-4.1 和 o3-mini），这些范式模拟了系统1（快速且直觉）或系统2（缓慢且深思熟虑）的思考方式，并将其与两个较小的模型（FinBERT-Prosus和FinBERT-Tone，后者在金融情感分析上进行了微调）进行基准比较。我们的研究结果表明，在这项任务中，无论是通过提示还是内置的设计，推理并不能提高性能。令人惊讶的是，最准确且与人类判断一致的模型方法组合是GPT-4o，没有使用任何链式思维（CoT）提示。 我们进一步探讨了语言复杂性和标注一致性水平对性能的影响，发现推理可能会导致过度思考，从而产生次优预测。这表明，在金融情感分类中，“系统1”式的快速、直觉性思考与人类判断更为契合，而由推理模型或CoT提示模拟的“系统2”式缓慢且深思熟虑的推理则不然。 我们的研究结果挑战了默认假设：更多的推理总是能带来更好的LLM决策，尤其是在高风险金融应用中。
URL
https://arxiv.org/abs/2506.04574
PDF
-
Multi-Domain ABSA Conversation Dataset Generation via LLMs for Real-World Evaluation and Model Comparison
2025-05-30 15:24:17Tejul Pandit, Meet Raval, Dhvani UpadhyayAbstract
Aspect-Based Sentiment Analysis (ABSA) offers granular insights into opinions but often suffers from the scarcity of diverse, labeled datasets that reflect real-world conversational nuances. This paper presents an approach for generating synthetic ABSA data using Large Language Models (LLMs) to address this gap. We detail the generation process aimed at producing data with consistent topic and sentiment distributions across multiple domains using GPT-4o. The quality and utility of the generated data were evaluated by assessing the performance of three state-of-the-art LLMs (Gemini 1.5 Pro, Claude 3.5 Sonnet, and DeepSeek-R1) on topic and sentiment classification tasks. Our results demonstrate the effectiveness of the synthetic data, revealing distinct performance trade-offs among the models: DeepSeekR1 showed higher precision, Gemini 1.5 Pro and Claude 3.5 Sonnet exhibited strong recall, and Gemini 1.5 Pro offered significantly faster inference. We conclude that LLM-based synthetic data generation is a viable and flexible method for creating valuable ABSA resources, facilitating research and model evaluation without reliance on limited or inaccessible real-world labeled data.
Abstract (translated)
基于方面的情感分析（ABSA）提供了对意见的精细洞察，但通常会因为缺乏多样且标注完善的现实对话数据而受限。本文提出了一种使用大型语言模型（LLMs）生成合成ABSA数据的方法，以弥补这一缺口。我们详细描述了利用GPT-4o来产生在多个领域内具有一致主题和情感分布的合成数据的过程。通过评估三种最先进的LLM（Gemini 1.5 Pro、Claude 3.5 Sonnet 和 DeepSeek-R1）在主题分类和情感分析任务上的性能，我们对生成的数据质量进行了评价。我们的结果显示了合成数据的有效性，并揭示了不同模型之间的性能权衡：DeepSeekR1表现出更高的精度，Gemini 1.5 Pro和Claude 3.5 Sonnet显示出强大的召回率，而Gemini 1.5 Pro则提供了显著更快的推理速度。我们得出结论，基于LLM的合成数据生成是一种可行且灵活的方法，能够创建有价值的ABSA资源，从而在没有受限或难以获取的真实世界标注数据的情况下支持研究和模型评估。
URL
https://arxiv.org/abs/2505.24701
PDF
-
On VLMs for Diverse Tasks in Multimodal Meme Classification
2025-05-27 09:25:46Deepesh Gavit, Debajyoti Mazumder, Samiran Das, Jasabanta PatroAbstract
In this paper, we present a comprehensive and systematic analysis of vision-language models (VLMs) for disparate meme classification tasks. We introduced a novel approach that generates a VLM-based understanding of meme images and fine-tunes the LLMs on textual understanding of the embedded meme text for improving the performance. Our contributions are threefold: (1) Benchmarking VLMs with diverse prompting strategies purposely to each sub-task; (2) Evaluating LoRA fine-tuning across all VLM components to assess performance gains; and (3) Proposing a novel approach where detailed meme interpretations generated by VLMs are used to train smaller language models (LLMs), significantly improving classification. The strategy of combining VLMs with LLMs improved the baseline performance by 8.34%, 3.52% and 26.24% for sarcasm, offensive and sentiment classification, respectively. Our results reveal the strengths and limitations of VLMs and present a novel strategy for meme understanding.
Abstract (translated)
在这篇论文中，我们对用于不同梗图分类任务的视觉-语言模型（VLM）进行了全面和系统的分析。我们提出了一种新颖的方法，该方法通过基于VLM的理解生成梗图图像，并在嵌入式文本理解上微调大型语言模型（LLMs），以提高性能。我们的贡献有三方面：(1) 使用多种针对每个子任务的提示策略对VLM进行基准测试；(2) 在所有VLM组件中评估LoRA微调带来的性能提升；以及(3) 提出了一种新颖的方法，其中通过VLM生成的详细梗图解释来训练小型语言模型（LLMs），从而显著提高了分类效果。结合使用VLM和LLM的战略分别在讽刺、冒犯性和情感分类任务上将基线性能提升了8.34%、3.52% 和 26.24%。我们的结果揭示了VLM的优势和局限性，并提出了一种理解梗图的新策略。
URL
https://arxiv.org/abs/2505.20937
PDF
-
ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification
2025-05-26 10:58:13Mohamed Masry, Mohamed Amen, Mohamed Elzyat, Mohamed Hamed, Norhan Magdy, Maram KhaledAbstract
Decoding natural language from brain activity using non-invasive electroencephalography (EEG) remains a significant challenge in neuroscience and machine learning, particularly for open-vocabulary scenarios where traditional methods struggle with noise and variability. Previous studies have achieved high accuracy on small-closed vocabularies, but it still struggles on open vocabularies. In this study, we propose ETS, a framework that integrates EEG with synchronized eye-tracking data to address two critical tasks: (1) open-vocabulary text generation and (2) sentiment classification of perceived language. Our model achieves a superior performance on BLEU and Rouge score for EEG-To-Text decoding and up to 10% F1 score on EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for high performance open vocabulary eeg-to-text system.
Abstract (translated)
使用非侵入性脑电图（EEG）从大脑活动解码自然语言仍然是神经科学和机器学习领域的一大挑战，特别是在开放词汇场景中，传统方法在处理噪声和变异方面存在困难。以往的研究虽然在小型封闭词汇表上实现了高精度，但在开放词汇表上的表现仍然不尽如人意。在这项研究中，我们提出了ETS框架，该框架结合了同步的眼动追踪数据与EEG信号，旨在解决两个关键任务：（1）开放式文本生成和（2）感知语言的情感分类。我们的模型在基于EEG的文本到语音解码上取得了BLEU和Rouge评分的卓越成绩，并且在基于EEG的情绪三分类任务中达到了高达10%的F1得分，显著优于监督基线模型。此外，我们还展示了所提出模型能够处理来自不同受试者和来源的数据，显示了其构建高性能开放词汇表EEG到文本系统的巨大潜力。
URL
https://arxiv.org/abs/2506.14783
PDF
-
Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification
2025-05-26 10:01:24Akram Elbouanani, Evan Dufraisse, Adrian PopescuAbstract
Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts.
Abstract (translated)
大型语言模型（LLM）中编码的政治偏见可能对下游应用产生负面影响。现有的偏见分析方法依赖于小型中间任务（如问卷回答或政治内容生成），并且依赖于LLMs本身进行分析，从而传播了偏见。我们提出了一种新方法，利用观察到的LLM情感预测在句子中的目标实体变化这一现象，并定义了一个基于熵的不一致性指标来编码这种预测变动性。 为了测试这种方法，我们在450条政治语句中插入了1319个具有不同人口统计学和政治背景的政治家名字，并使用六种广泛使用的语言中的七种模型进行了目标导向的情感预测。我们观察到所有测试组合中存在不一致性，并在不同的粒度级别上进行了一致的统计分析。 我们发现对左翼和极右翼政客有积极和消极的偏见，且具有相似政治立场的政治家之间存在正相关性。西方语言中的偏见强度高于其他语言。较大的模型表现出更强且更一致的偏见，并减少了类似语言之间的差异。 为了部分缓解LLM在目标导向情感分类（TSC）中的不可靠性，我们用虚构但可信的替身替代了政治家的名字。
URL
https://arxiv.org/abs/2505.19776
PDF
-
The Role of Diversity in In-Context Learning for Large Language Models
2025-05-26 02:37:26Wenyang Xiao, Haoyu Zhao, Lingxiao HuangAbstract
In-context learning (ICL) is a crucial capability of current large language models (LLMs), where the selection of examples plays a key role in performance. While most existing approaches focus on selecting the most similar examples to the query, the impact of diversity in example selection remains underexplored. We systematically investigate the role of diversity in in-context example selection through experiments across a range of tasks, from sentiment classification to more challenging math and code problems. Experiments on Llama-3.1, Gemma-2, and Mistral-v0.3 families of models show that diversity-aware selection methods improve performance, particularly on complex tasks like math and code, and enhance robustness to out-of-distribution queries. To support these findings, we introduce a theoretical framework that explains the benefits of incorporating diversity in in-context example selection.
Abstract (translated)
上下文学习（ICL）是当前大型语言模型（LLM）的一项关键能力，其中选择示例在性能中起着重要作用。虽然大多数现有方法侧重于选取与查询最相似的示例，但多样性的选择对性能的影响仍鲜有探索。我们通过一系列实验系统地研究了多样性在上下文示例选择中的作用，涵盖了从情感分类到更具挑战性的数学和编程问题等各种任务。在Llama-3.1、Gemma-2和Mistral-v0.3系列模型上的实验证明，具有多样性的示例选择方法能够提高性能，特别是在复杂的数学和编程等任务上，并且还能增强对分布外查询的鲁棒性。为了支持这些发现，我们引入了一个理论框架来解释在上下文示例选择中融入多样性的好处。
URL
https://arxiv.org/abs/2505.19426
PDF
-
CrosGrpsABS: Cross-Attention over Syntactic and Semantic Graphs for Aspect-Based Sentiment Analysis in a Low-Resource Language
2025-05-25 07:42:32Md. Mithun Hossain, Md. Shakil Hossain, Sudipto Chaki, Md. Rajib Hossain, Md. Saifur Rahman, A. B. M. Shawkat AliAbstract
Aspect-Based Sentiment Analysis (ABSA) is a fundamental task in natural language processing, offering fine-grained insights into opinions expressed in text. While existing research has largely focused on resource-rich languages like English which leveraging large annotated datasets, pre-trained models, and language-specific tools. These resources are often unavailable for low-resource languages such as Bengali. The ABSA task in Bengali remains poorly explored and is further complicated by its unique linguistic characteristics and a lack of annotated data, pre-trained models, and optimized hyperparameters. To address these challenges, this research propose CrosGrpsABS, a novel hybrid framework that leverages bidirectional cross-attention between syntactic and semantic graphs to enhance aspect-level sentiment classification. The CrosGrpsABS combines transformerbased contextual embeddings with graph convolutional networks, built upon rule-based syntactic dependency parsing and semantic similarity computations. By employing bidirectional crossattention, the model effectively fuses local syntactic structure with global semantic context, resulting in improved sentiment classification performance across both low- and high-resource settings. We evaluate CrosGrpsABS on four low-resource Bengali ABSA datasets and the high-resource English SemEval 2014 Task 4 dataset. The CrosGrpsABS consistently outperforms existing approaches, achieving notable improvements, including a 0.93% F1-score increase for the Restaurant domain and a 1.06% gain for the Laptop domain in the SemEval 2014 Task 4 benchmark.
Abstract (translated)
基于方面的情感分析（ABSA）是自然语言处理中的一个基本任务，它为文本中表达的意见提供了细致的洞察。尽管现有研究主要集中在资源丰富的语言上，如英语，并且依赖于大规模标注数据集、预训练模型和特定的语言工具，这些资源对于像孟加拉语这样的低资源语言往往是不可用的。孟加拉语中的ABSA任务尚未充分探索，并且由于其独特的语言特征以及缺乏标注数据、预训练模型和优化的超参数而变得更加复杂。 为了应对这些挑战，这项研究提出了CrosGrpsABS，这是一种新颖的混合框架，利用句法图与语义图之间的双向交叉注意来增强方面级情感分类。CrosGrpsABS结合了基于转换器的上下文嵌入和图形卷积网络，并建立在基于规则的句法依存分析和语义相似性计算之上。通过使用双向交叉注意力机制，该模型有效地融合了局部句法结构与全局语义背景信息，在低资源环境和高资源环境中均提高了情感分类性能。 我们在四个孟加拉语低资源ABSA数据集以及高资源英语SemEval 2014 Task 4数据集上对CrosGrpsABS进行了评估。在所有评测中，CrosGrpsABS始终优于现有方法，并实现了显著的改进，包括在SemEval 2014 Task 4基准测试中的餐厅领域和笔记本电脑领域的F1得分分别提高了0.93%和1.06%。
URL
https://arxiv.org/abs/2505.19018
PDF
-
DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer
2025-05-21 04:20:30Sona Elza Simon, Preethi JyothiAbstract
Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model's parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X, a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.
Abstract (translated)
有效的跨语言迁移仍然是扩大大型语言模型从高资源语言到低资源语言效益的关键挑战。为了解决这个问题，先前的研究探索了多种方法来结合任务特定数据（在高资源源语言中）中的任务知识和未标记文本（在低资源目标语言中）中的语言知识。其中一种值得注意的方法是提出了可组合的稀疏微调(SFT)，用于跨语言迁移，这种方法学习任务特定和语言特定的稀疏掩码来选择预训练模型参数的一个子集进行进一步的微调。这些稀疏微调向量（SFTs）随后与预训练模型结合使用，以促进仅通过源语言的任务特定数据就实现零样本跨语言迁移到目标语言中的任务。用于SFTs的稀疏掩码是通过简单的基于幅度的剪枝来识别的。 在我们的工作中，我们引入了DeFT-X，这是一种新的可组合SFT方法，在进行幅度剪枝之前使用奇异值分解对预训练模型的权重矩阵进行去噪处理，从而产生更健壮的SFTs。我们在多样化的极度低资源语言（用于情感分类NusaX和自然语言推理AmericasNLI）上评估了DeFT-X，并证明它在性能上与SFT和其他显着的跨语言迁移基线相当或优于它们。
URL
https://arxiv.org/abs/2505.15090
PDF
-
Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications
2025-05-20 21:12:58Fadel M. Megahed, Ying-Ju Chen, L. Allision Jones-Farmer, Younghwa Lee, Jiawei Brooke Wang, Inez M. ZwetslootAbstract
This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification, addressing the lack of established reliability assessment methods. Adapting psychometric principles, we determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability. Our case study examines financial news sentiment classification across 14 LLMs (including claude-3-7-sonnet, gpt-4o, deepseek-r1, gemma3, llama3.2, phi4, and command-r-plus), with five replicates per model on 1,350 articles. Models demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with minimal differences between expensive and economical models from the same families. When validated against StockNewsAPI labels, models achieved strong performance (accuracy 0.76-0.88), with smaller models like gemma3:1B, llama3.2:3B, and claude-3-5-haiku outperforming larger counterparts. All models performed at chance when predicting actual market movements, indicating task constraints rather than model limitations. Our framework provides systematic guidance for LLM selection, sample size planning, and reliability assessment, enabling organizations to optimize resources for classification tasks.
Abstract (translated)
这项研究提出了一种评估大型语言模型（LLM）二元文本分类一致性的框架，旨在解决现有可靠性评估方法不足的问题。通过借鉴心理测量学原理，我们确定了样本量的需求，开发了无效响应的度量标准，并对评分者内和评分者间的可靠性进行了评价。我们的案例研究考察了14个大型语言模型（包括claude-3-7-sonnet、gpt-4o、deepseek-r1、gemma3、llama3.2、phi4和command-r-plus）在财务新闻情绪分类方面的表现，每个模型重复进行五次测试，共分析了1,350篇文章。各模型展示了较高的评分者内一致性，在90%-98%的样本中实现了完全一致的意见，并且来自同一家族的昂贵型和经济型模型之间差异很小。当与StockNewsAPI标签验证时，这些模型表现出色（准确率在0.76到0.88之间），其中较小规模的模型如gemma3:1B、llama3.2:3B以及claude-3-5-haiku的表现优于其大型同类。所有模型在预测实际市场走势时表现均接近随机水平，这表明任务难度而非模型能力限制了性能。我们的框架为LLM选择、样本量规划和可靠性评估提供了系统性指导，帮助组织优化资源以应对分类任务的挑战。
URL
https://arxiv.org/abs/2505.14918
PDF
-
PL-FGSA: A Prompt Learning Framework for Fine-Grained Sentiment Analysis Based on MindSpore
2025-05-20 10:18:10Zhenkai Qin, Jiajing He, Qiao FangAbstract
Fine-grained sentiment analysis (FGSA) aims to identify sentiment polarity toward specific aspects within a text, enabling more precise opinion mining in domains such as product reviews and social media. However, traditional FGSA approaches often require task-specific architectures and extensive annotated data, limiting their generalization and scalability. To address these challenges, we propose PL-FGSA, a unified prompt learning-based framework implemented using the MindSpore platform, which integrates prompt design with a lightweight TextCNN backbone. Our method reformulates FGSA as a multi-task prompt-augmented generation problem, jointly tackling aspect extraction, sentiment classification, and causal explanation in a unified paradigm. By leveraging prompt-based guidance, PL-FGSA enhances interpretability and achieves strong performance under both full-data and low-resource conditions. Experiments on three benchmark datasets-SST-2, SemEval-2014 Task 4, and MAMS-demonstrate that our model consistently outperforms traditional fine-tuning methods and achieves F1-scores of 0.922, 0.694, and 0.597, respectively. These results validate the effectiveness of prompt-based generalization and highlight the practical value of PL-FGSA for real-world sentiment analysis tasks.
Abstract (translated)
细粒度情感分析（FGSA）旨在识别文本中针对特定方面的情感极性，从而在产品评论和社会媒体等领域实现更精确的意见挖掘。然而，传统的FGSA方法通常需要特定任务的架构和大量的注释数据，这限制了它们的泛化能力和可扩展性。为了解决这些挑战，我们提出了PL-FGSA，这是一个基于提示学习的统一框架，使用MindSpore平台构建，并结合了轻量级的TextCNN骨干网络。我们的方法将FGSA重新表述为一个多任务提示增强生成问题，在一个统一的范式中同时处理方面提取、情感分类和因果解释。通过利用基于提示的指导，PL-FGSA增强了可解释性，并在全数据集和低资源条件下均表现出色。 我们在三个基准数据集上进行了实验——SST-2、SemEval-2014 Task 4 和 MAMS，结果表明我们的模型始终优于传统的微调方法，并分别取得了F1分数为0.922、0.694 和 0.597 的成绩。这些结果验证了基于提示泛化的有效性，并突显了PL-FGSA在现实世界情感分析任务中的实际价值。
URL
https://arxiv.org/abs/2505.14165
PDF
