Paper Reading AI Learner

BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models

2026-02-21 10:43:07
Thura Aung, Jann Railey Montalan, Jian Gang Ngui, Peerat Limkonchotiwat

Abstract

We introduce BURMESE-SAN, the first holistic benchmark that systematically evaluates large language models (LLMs) for Burmese across three core NLP competencies: understanding (NLU), reasoning (NLR), and generation (NLG). BURMESE-SAN consolidates seven subtasks spanning these competencies, including Question Answering, Sentiment Analysis, Toxicity Detection, Causal Reasoning, Natural Language Inference, Abstractive Summarization, and Machine Translation, several of which were previously unavailable for Burmese. The benchmark is constructed through a rigorous native-speaker-driven process to ensure linguistic naturalness, fluency, and cultural authenticity while minimizing translation-induced artifacts. We conduct a large-scale evaluation of both open-weight and commercial LLMs to examine challenges in Burmese modeling arising from limited pretraining coverage, rich morphology, and syntactic variation. Our results show that Burmese performance depends more on architectural design, language representation, and instruction tuning than on model scale alone. In particular, Southeast Asia regional fine-tuning and newer model generations yield substantial gains. Finally, we release BURMESE-SAN as a public leaderboard to support systematic evaluation and sustained progress in Burmese and other low-resource languages. this https URL

Abstract (translated)

我们介绍了BURMESE-SAN,这是第一个全面的基准测试框架,用于系统地评估针对缅甸语的大规模语言模型(LLMs)在三个核心自然语言处理能力方面:理解(NLU)、推理(NLR)和生成(NLG)。BURMESE-SAN整合了七个涵盖这些能力的任务,包括问题回答、情感分析、毒性检测、因果推理、自然语言推断、摘要生成以及机器翻译。其中一些任务之前在缅甸语中并未提供。 该基准通过严格的母语者驱动的过程构建而成,以确保语言的自然性、流畅性和文化真实性,同时尽量减少因翻译而产生的误差。我们对开源和商业LLMs进行了大规模评估,旨在考察由于预训练覆盖率不足、丰富的词形变化以及句法差异导致的缅甸语建模挑战。我们的结果显示,在模型设计、语言表示和指令微调方面,缅甸语的表现依赖性高于单纯的模型规模大小。尤其是东南亚地区的细化调整及较新的模型版本可以带来显著改进。 最后,我们发布BURMESE-SAN作为公共排行榜,以支持系统性的评估并促进包括低资源语言在内的缅语的持续进步。(详情请参见:[原文链接]) 注意:末尾的"[原文链接]"应替换为实际提供的URL。

URL

https://arxiv.org/abs/2602.18788

PDF

https://arxiv.org/pdf/2602.18788.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot