Abstract
Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform a comprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (approximately 20-100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP, attention should be paid to both context length and, in particular, instance count.
Abstract (translated)
用户通常依赖大语言模型(LLMs)处理多份文档或对多个实例进行分析。例如,分析若干影评的整体情感需要LLM逐一处理每条评论的情感,以提供最终的汇总答案。尽管LLM在单任务上的表现通常良好,但关于其处理多实例输入时的性能研究较少。本文针对LLM在单任务中表现优异的场景,对其多实例处理(MIP)能力进行了全面评估。结果表明,所有LLM均呈现轻微性能下降的模式(实例数约20-100个),随后在实例数量更大时出现性能崩溃。关键的是,我们的分析显示:虽然上下文长度与这种性能下降相关,但实例数量对最终结果的影响更为显著。这一发现表明,在优化LLM的MIP性能时,应同时关注上下文长度,尤其需重视实例数量。
URL
https://arxiv.org/abs/2603.22608