Abstract
Large Language Models (LLMs) are rapidly transitioning from static Natural Language Processing (NLP) tasks including sentiment analysis and event extraction to acting as dynamic decision-making agents in complex financial environments. However, the evolution of LLMs into autonomous financial agents faces a significant dilemma in evaluation paradigms. Direct live trading is irreproducible and prone to outcome bias by confounding luck with skill, whereas existing static benchmarks are often confined to entity-level stock picking and ignore broader market attention. To facilitate the rigorous analysis of these challenges, we introduce CN-Buzz2Portfolio, a reproducible benchmark grounded in the Chinese market that maps daily trending news to macro and sector asset allocation. Spanning a rolling horizon from 2024 to mid-2025, our dataset simulates a realistic public attention stream, requiring agents to distill investment logic from high-exposure narratives instead of pre-filtered entity news. We propose a Tri-Stage CPA Agent Workflow involving Compression, Perception, and Allocation to evaluate LLMs on broad asset classes such as Exchange Traded Funds (ETFs) rather than individual stocks, thereby reducing idiosyncratic volatility. Extensive experiments on nine LLMs reveal significant disparities in how models translate macro-level narratives into portfolio weights. This work provides new insights into the alignment between general reasoning and financial decision-making, and all data, codes, and experiments are released to promote sustainable financial agent research.
Abstract (translated)
大语言模型(LLMs)正迅速从静态自然语言处理(NLP)任务(如情感分析和事件抽取)转向在复杂金融环境中充当动态决策代理。然而,LLMs向自主金融代理的演进在评估范式方面面临显著困境:直接实时交易不可复现,且易将运气与技能混淆而产生结果偏差;而现有静态基准通常局限于个股层面的选股,忽视了更广泛的市场关注。为促进对这些挑战的严谨分析,我们推出了CN-Buzz2Portfolio——一个基于中国市场的可复现基准,它将每日趋势新闻映射为宏观及行业资产配置。该数据集涵盖2024年至2025年年中的滚动时间窗口,模拟了真实的公众关注流,要求代理从高曝光度叙事中提炼投资逻辑,而非依赖预筛选的个股新闻。我们提出了包含压缩、感知与分配三阶段的CPA代理工作流,以评估大语言模型在交易所交易基金(ETFs)等广义资产类别上的表现,从而降低个体波动性。对九个大语言模型的广泛实验揭示了不同模型将宏观叙事转化为投资组合权重的显著差异。本研究为通用推理与金融决策之间的对齐提供了新见解,所有数据、代码和实验均已公开,以推动可持续的金融代理研究。
URL
https://arxiv.org/abs/2603.22305