Abstract
Large Language Models (LLMs) are increasingly deployed in socially sensitive domains, yet their unpredictable behaviors, ranging from misaligned intent to inconsistent personality, pose significant risks. We introduce SteerEval, a hierarchical benchmark for evaluating LLM controllability across three domains: language features, sentiment, and personality. Each domain is structured into three specification levels: L1 (what to express), L2 (how to express), and L3 (how to instantiate), connecting high-level behavioral intent to concrete textual output. Using SteerEval, we systematically evaluate contemporary steering methods, revealing that control often degrades at finer-grained levels. Our benchmark offers a principled and interpretable framework for safe and controllable LLM behavior, serving as a foundation for future research.
Abstract (translated)
大型语言模型(LLMs)在社会敏感领域中的部署日益增多,但它们不可预测的行为——从意图不一致到个性不统一——带来了显著的风险。我们引入了SteerEval,这是一个分层基准,用于评估LLM在其三个领域的可控性:语言特征、情感和个性。每个领域都分为三个规范层级:L1(要表达什么)、L2(如何表达)和L3(如何实例化),将高层次的行为意图与具体的文本输出联系起来。通过使用SteerEval,我们系统地评估了当前的导向方法,揭示出在更细粒度层次上控制往往有所下降。我们的基准测试提供了一个原则性和可解释性的框架,以实现安全且可控的LLM行为,并为未来的研究奠定了基础。
URL
https://arxiv.org/abs/2603.02578