Abstract
In this paper, we present a comprehensive and systematic analysis of vision-language models (VLMs) for disparate meme classification tasks. We introduced a novel approach that generates a VLM-based understanding of meme images and fine-tunes the LLMs on textual understanding of the embedded meme text for improving the performance. Our contributions are threefold: (1) Benchmarking VLMs with diverse prompting strategies purposely to each sub-task; (2) Evaluating LoRA fine-tuning across all VLM components to assess performance gains; and (3) Proposing a novel approach where detailed meme interpretations generated by VLMs are used to train smaller language models (LLMs), significantly improving classification. The strategy of combining VLMs with LLMs improved the baseline performance by 8.34%, 3.52% and 26.24% for sarcasm, offensive and sentiment classification, respectively. Our results reveal the strengths and limitations of VLMs and present a novel strategy for meme understanding.
Abstract (translated)
在这篇论文中，我们对用于不同梗图分类任务的视觉-语言模型（VLM）进行了全面和系统的分析。我们提出了一种新颖的方法，该方法通过基于VLM的理解生成梗图图像，并在嵌入式文本理解上微调大型语言模型（LLMs），以提高性能。我们的贡献有三方面：(1) 使用多种针对每个子任务的提示策略对VLM进行基准测试；(2) 在所有VLM组件中评估LoRA微调带来的性能提升；以及(3) 提出了一种新颖的方法，其中通过VLM生成的详细梗图解释来训练小型语言模型（LLMs），从而显著提高了分类效果。结合使用VLM和LLM的战略分别在讽刺、冒犯性和情感分类任务上将基线性能提升了8.34%、3.52% 和 26.24%。我们的结果揭示了VLM的优势和局限性，并提出了一种理解梗图的新策略。
URL
https://arxiv.org/abs/2505.20937
