Abstract
Machine learning methods are increasingly applied to analyze health-related public discourse based on large-scale data, but questions remain regarding their ability to accurately detect different types of health sentiments. Especially, Large Language Models (LLMs) have gained attention as a powerful technology, yet their accuracy and feasibility in capturing different opinions and perspectives on health issues are largely unexplored. Thus, this research examines how accurate the three prominent LLMs (GPT, Gemini, and LLAMA) are in detecting risk-promoting versus health-supporting sentiments across two critical public health topics: Human Papillomavirus (HPV) vaccination and heated tobacco products (HTPs). Drawing on data from Facebook and Twitter, we curated multiple sets of messages supporting or opposing recommended health behaviors, supplemented with human annotations as the gold standard for sentiment classification. The findings indicate that all three LLMs generally demonstrate substantial accuracy in classifying risk-promoting and health-supporting sentiments, although notable discrepancies emerge by platform, health issue, and model type. Specifically, models often show higher accuracy for risk-promoting sentiment on Facebook, whereas health-supporting messages on Twitter are more accurately detected. An additional analysis also shows the challenges LLMs face in reliably detecting neutral messages. These results highlight the importance of carefully selecting and validating language models for public health analyses, particularly given potential biases in training data that may lead LLMs to overestimate or underestimate the prevalence of certain perspectives.
Abstract (translated)
基于大规模数据的机器学习方法越来越多地被应用于分析与健康相关的公众言论,但关于它们准确检测不同类型健康情感的能力仍然存在疑问。特别是,大型语言模型(LLMs)作为一项强大的技术引起了关注,然而它们在捕捉健康问题的不同观点和视角方面的准确性及可行性尚未得到充分探索。因此,这项研究考察了三种知名的大规模语言模型(GPT、Gemini 和 LLAMA),它们在检测与两种关键公共卫生话题相关的情感方面——人类乳头瘤病毒(HPV)疫苗接种和加热烟草产品(HTPs)——的准确度如何。我们从 Facebook 和 Twitter 中提取了多组支持或反对推荐健康行为的信息,并通过人工注释作为情感分类的标准来补充这些数据。研究结果表明,这三种 LLM 通常在识别风险促进与健康支持性情感方面表现出相当高的准确性,尽管这种准确性因平台、健康问题和模型类型而异。具体而言,模型往往在 Facebook 上对风险促进的情感检测更准确,而在 Twitter 上则更擅长于检测健康支持性的信息。此外的分析还显示了 LLM 在可靠地检测中立消息时面临的挑战。这些结果突显了在进行公共卫生分析时选择和验证语言模型的重要性,尤其是在考虑到训练数据中存在的潜在偏见可能导致 LLM 对某些观点的流行程度估计过高或过低的情况下。
URL
https://arxiv.org/abs/2507.04364