Paper Reading AI Learner

Large Language Models' Varying Accuracy in Recognizing Risk-Promoting and Health-Supporting Sentiments in Public Health Discourse: The Cases of HPV Vaccination and Heated Tobacco Products

2025-07-06 11:57:02
Soojong Kim, Kwanho Kim, Hye Min Kim

Abstract

Machine learning methods are increasingly applied to analyze health-related public discourse based on large-scale data, but questions remain regarding their ability to accurately detect different types of health sentiments. Especially, Large Language Models (LLMs) have gained attention as a powerful technology, yet their accuracy and feasibility in capturing different opinions and perspectives on health issues are largely unexplored. Thus, this research examines how accurate the three prominent LLMs (GPT, Gemini, and LLAMA) are in detecting risk-promoting versus health-supporting sentiments across two critical public health topics: Human Papillomavirus (HPV) vaccination and heated tobacco products (HTPs). Drawing on data from Facebook and Twitter, we curated multiple sets of messages supporting or opposing recommended health behaviors, supplemented with human annotations as the gold standard for sentiment classification. The findings indicate that all three LLMs generally demonstrate substantial accuracy in classifying risk-promoting and health-supporting sentiments, although notable discrepancies emerge by platform, health issue, and model type. Specifically, models often show higher accuracy for risk-promoting sentiment on Facebook, whereas health-supporting messages on Twitter are more accurately detected. An additional analysis also shows the challenges LLMs face in reliably detecting neutral messages. These results highlight the importance of carefully selecting and validating language models for public health analyses, particularly given potential biases in training data that may lead LLMs to overestimate or underestimate the prevalence of certain perspectives.

Abstract (translated)

基于大规模数据的机器学习方法越来越多地被应用于分析与健康相关的公众言论,但关于它们准确检测不同类型健康情感的能力仍然存在疑问。特别是,大型语言模型(LLMs)作为一项强大的技术引起了关注,然而它们在捕捉健康问题的不同观点和视角方面的准确性及可行性尚未得到充分探索。因此,这项研究考察了三种知名的大规模语言模型(GPT、Gemini 和 LLAMA),它们在检测与两种关键公共卫生话题相关的情感方面——人类乳头瘤病毒(HPV)疫苗接种和加热烟草产品(HTPs)——的准确度如何。我们从 Facebook 和 Twitter 中提取了多组支持或反对推荐健康行为的信息,并通过人工注释作为情感分类的标准来补充这些数据。研究结果表明,这三种 LLM 通常在识别风险促进与健康支持性情感方面表现出相当高的准确性,尽管这种准确性因平台、健康问题和模型类型而异。具体而言,模型往往在 Facebook 上对风险促进的情感检测更准确,而在 Twitter 上则更擅长于检测健康支持性的信息。此外的分析还显示了 LLM 在可靠地检测中立消息时面临的挑战。这些结果突显了在进行公共卫生分析时选择和验证语言模型的重要性,尤其是在考虑到训练数据中存在的潜在偏见可能导致 LLM 对某些观点的流行程度估计过高或过低的情况下。

URL

https://arxiv.org/abs/2507.04364

PDF

https://arxiv.org/pdf/2507.04364.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot