Abstract
One fundamental question for the social sciences today is: how much can we trust highly complex predictive models like ChatGPT? This study tests the hypothesis that subtle changes in the structure of prompts do not produce significant variations in the classification results of sentiment polarity analysis generated by the Large Language Model GPT-4o mini. Using a dataset of 100.000 comments in Spanish on four Latin American presidents, the model classified the comments as positive, negative, or neutral on 10 occasions, varying the prompts slightly each time. The experimental methodology included exploratory and confirmatory analyses to identify significant discrepancies among classifications. The results reveal that even minor modifications to prompts such as lexical, syntactic, or modal changes, or even their lack of structure impact the classifications. In certain cases, the model produced inconsistent responses, such as mixing categories, providing unsolicited explanations, or using languages other than Spanish. Statistical analysis using Chi-square tests confirmed significant differences in most comparisons between prompts, except in one case where linguistic structures were highly similar. These findings challenge the robustness and trust of Large Language Models for classification tasks, highlighting their vulnerability to variations in instructions. Moreover, it was evident that the lack of structured grammar in prompts increases the frequency of hallucinations. The discussion underscores that trust in Large Language Models is based not only on technical performance but also on the social and institutional relationships underpinning their use.
Abstract (translated)
今天社会科学面临的一个基本问题是：我们能在多大程度上信任像ChatGPT这样高度复杂的预测模型？这项研究检验了这样一个假设，即提示结构的细微变化不会导致大型语言模型（如GPT-4o mini）生成的情感极性分析分类结果产生显著差异。该研究使用了一组10万条西班牙语评论数据集，这些评论针对四个拉丁美洲总统。实验中，模型在10个不同场景下对这些建议进行了积极、消极或中立的分类，并且每次提示都有细微的不同之处。 实验方法包括了探索性和确认性分析，以识别分类之间的显著差异。研究结果显示，即使是词汇、句法和模态上的微小变化，或者缺乏结构化都会影响模型的分类结果。在某些情况下，该模型产生了不一致的回答，例如混淆类别、提供非请求解释或使用西班牙语以外的语言。 卡方检验的统计分析确认了大多数提示比较之间的显著差异，唯一的例外是语言结构高度相似的情况。这些发现挑战了大型语言模型在分类任务中的稳健性和可信度，并突显了它们对指令变化的脆弱性。此外还观察到，缺乏结构化语法的提示会增加幻觉现象的发生频率。 讨论强调了信任大型语言模型不仅基于其技术性能，还需要考虑在其使用中所涉及的社会和机构关系。
URL
https://arxiv.org/abs/2504.12180
