Abstract
This paper explores the development of a multimodal sentiment analysis model that integrates text, audio, and visual data to enhance sentiment classification. The goal is to improve emotion detection by capturing the complex interactions between these modalities, thereby enabling more accurate and nuanced sentiment interpretation. The study evaluates three feature fusion strategies -- late stage fusion, early stage fusion, and multi-headed attention -- within a transformer-based architecture. Experiments were conducted using the CMU-MOSEI dataset, which includes synchronized text, audio, and visual inputs labeled with sentiment scores. Results show that early stage fusion significantly outperforms late stage fusion, achieving an accuracy of 71.87\%, while the multi-headed attention approach offers marginal improvement, reaching 72.39\%. The findings suggest that integrating modalities early in the process enhances sentiment classification, while attention mechanisms may have limited impact within the current framework. Future work will focus on refining feature fusion techniques, incorporating temporal data, and exploring dynamic feature weighting to further improve model performance.
Abstract (translated)
本文探讨了一种多模态情感分析模型的发展,该模型结合了文本、音频和视觉数据以增强情感分类。目标是通过捕捉这些模式之间的复杂交互来提高情绪检测的准确性,并实现更准确和细致的情感解读。研究评估了三种特征融合策略——晚期融合、早期融合和多头注意力机制——在基于变压器架构中的表现。实验使用CMU-MOSEI数据集进行,该数据集包括同步文本、音频和视觉输入,并附有情感评分标签。 实验结果显示,早期融合显著优于晚期融合,在准确性上达到了71.87%,而多头注意机制则提供了边际改进,准确率达到72.39%。研究发现表明,将模式在处理过程中尽早整合可以提升情感分类的效果,而在当前框架下注意力机制的影响有限。未来的研究将重点放在完善特征融合技术、纳入时间数据以及探索动态特征加权上,以进一步提高模型性能。
URL
https://arxiv.org/abs/2501.08085