Abstract
We present PersonaConvBench, a large-scale benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs). Unlike existing work that focuses on either personalization or conversational structure in isolation, PersonaConvBench integrates both, offering three core tasks: sentence classification, impact regression, and user-centric text generation across ten diverse Reddit-based domains. This design enables systematic analysis of how personalized conversational context shapes LLM outputs in realistic multi-user scenarios. We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements, including a 198 percent relative gain over the best non-conversational baseline in sentiment classification. By releasing PersonaConvBench with evaluations and code, we aim to support research on LLMs that adapt to individual styles, track long-term context, and produce contextually rich, engaging responses.
Abstract (translated)
我们介绍了PersonaConvBench，这是一个大规模基准测试系统，用于评估大型语言模型（LLM）在多轮对话中个性化推理和生成的能力。与现有研究仅专注于个人化或对话结构不同，PersonaConvBench将两者结合起来，并提供三项核心任务：句子分类、影响回归以及跨十个基于Reddit的多样化领域进行以用户为中心的文本生成。这种设计能够系统地分析个性化的对话背景如何塑造LLM在现实多用户场景中的输出结果。 我们在统一的提示设置下对几种商用和开源的大型语言模型进行了基准测试，并观察到将个性化历史记录纳入考量可带来显著性能提升，包括在情感分类上比最佳非对话基线高出198%的相对增益。通过发布PersonaConvBench及其评估代码，我们的目标是支持研究那些能够适应个人风格、追踪长期背景信息并生成丰富且引人入胜响应的大型语言模型的研究工作。
URL
https://arxiv.org/abs/2505.14106
