Abstract
Although the annotation paradigm based on Large Language Models (LLMs) has made significant breakthroughs in recent years, its actual deployment still has two core bottlenecks: first, the cost of calling commercial APIs in large-scale annotation is very expensive; second, in scenarios that require fine-grained semantic understanding, such as sentiment classification and toxicity classification, the annotation accuracy of LLMs is even lower than that of Small Language Models (SLMs) dedicated to this field. To address these problems, we propose a new paradigm of multi-model cooperative annotation and design a fully automatic annotation framework AutoAnnotator based on this. Specifically, AutoAnnotator consists of two layers. The upper-level meta-controller layer uses the generation and reasoning capabilities of LLMs to select SLMs for annotation, automatically generate annotation code and verify difficult samples; the lower-level task-specialist layer consists of multiple SLMs that perform annotation through multi-model voting. In addition, we use the difficult samples obtained by the secondary review of the meta-controller layer as the reinforcement learning set and fine-tune the SLMs in stages through a continual learning strategy, thereby improving the generalization of SLMs. Extensive experiments show that AutoAnnotator outperforms existing open-source/API LLMs in zero-shot, one-shot, CoT, and majority voting settings. Notably, AutoAnnotator reduces the annotation cost by 74.15% compared to directly annotating with GPT-3.5-turbo, while still improving the accuracy by 6.21%. Project page: this https URL.
Abstract (translated)
尽管基于大型语言模型（LLM）的注释范式近年来取得了重大突破，但在实际部署中仍然存在两个核心瓶颈：首先，在大规模标注场景下调用商业API的成本非常高昂；其次，在需要细粒度语义理解的情境下，例如情感分类和毒性分类等任务，LLM的标注准确率甚至低于专门为此领域设计的小型语言模型（SLMs）。为了解决这些问题，我们提出了一种多模态协同注释的新范式，并基于此设计了一个全自动注释框架AutoAnnotator。 具体来说，AutoAnnotator包含两个层次。顶层元控制器层利用LLM的生成和推理能力来选择合适的SLM进行标注，自动生成标注代码并验证难例；底层任务专家层则由多个专精于特定任务的SLM组成，通过多模型投票的方式执行注释工作。此外，我们还采用了元控制器层二次审查得到的难例作为强化学习的数据集，并通过持续学习策略分阶段微调SLMs，从而提高其泛化能力。 广泛的实验表明，在零样本、单样本、CoT（链式推理）和多数表决等设置下，AutoAnnotator的表现优于现有的开源/API LLM。特别值得注意的是，相较于直接使用GPT-3.5-turbo进行标注，AutoAnnotator将注释成本降低了74.15%，同时仍然提高了6.21%的准确性。 项目页面: [请参阅此链接](https://this.is.the.url/project-page/)（请注意替换实际URL）。
URL
https://arxiv.org/abs/2506.16393
