Abstract
We introduce BURMESE-SAN, the first holistic benchmark that systematically evaluates large language models (LLMs) for Burmese across three core NLP competencies: understanding (NLU), reasoning (NLR), and generation (NLG). BURMESE-SAN consolidates seven subtasks spanning these competencies, including Question Answering, Sentiment Analysis, Toxicity Detection, Causal Reasoning, Natural Language Inference, Abstractive Summarization, and Machine Translation, several of which were previously unavailable for Burmese. The benchmark is constructed through a rigorous native-speaker-driven process to ensure linguistic naturalness, fluency, and cultural authenticity while minimizing translation-induced artifacts. We conduct a large-scale evaluation of both open-weight and commercial LLMs to examine challenges in Burmese modeling arising from limited pretraining coverage, rich morphology, and syntactic variation. Our results show that Burmese performance depends more on architectural design, language representation, and instruction tuning than on model scale alone. In particular, Southeast Asia regional fine-tuning and newer model generations yield substantial gains. Finally, we release BURMESE-SAN as a public leaderboard to support systematic evaluation and sustained progress in Burmese and other low-resource languages. this https URL
Abstract (translated)
我们介绍了BURMESE-SAN,这是第一个全面的基准测试框架,用于系统地评估针对缅甸语的大规模语言模型(LLMs)在三个核心自然语言处理能力方面:理解(NLU)、推理(NLR)和生成(NLG)。BURMESE-SAN整合了七个涵盖这些能力的任务,包括问题回答、情感分析、毒性检测、因果推理、自然语言推断、摘要生成以及机器翻译。其中一些任务之前在缅甸语中并未提供。 该基准通过严格的母语者驱动的过程构建而成,以确保语言的自然性、流畅性和文化真实性,同时尽量减少因翻译而产生的误差。我们对开源和商业LLMs进行了大规模评估,旨在考察由于预训练覆盖率不足、丰富的词形变化以及句法差异导致的缅甸语建模挑战。我们的结果显示,在模型设计、语言表示和指令微调方面,缅甸语的表现依赖性高于单纯的模型规模大小。尤其是东南亚地区的细化调整及较新的模型版本可以带来显著改进。 最后,我们发布BURMESE-SAN作为公共排行榜,以支持系统性的评估并促进包括低资源语言在内的缅语的持续进步。(详情请参见:[原文链接]) 注意:末尾的"[原文链接]"应替换为实际提供的URL。
URL
https://arxiv.org/abs/2602.18788