Abstract
Decoding natural language from brain activity using non-invasive electroencephalography (EEG) remains a significant challenge in neuroscience and machine learning, particularly for open-vocabulary scenarios where traditional methods struggle with noise and variability. Previous studies have achieved high accuracy on small-closed vocabularies, but it still struggles on open vocabularies. In this study, we propose ETS, a framework that integrates EEG with synchronized eye-tracking data to address two critical tasks: (1) open-vocabulary text generation and (2) sentiment classification of perceived language. Our model achieves a superior performance on BLEU and Rouge score for EEG-To-Text decoding and up to 10% F1 score on EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for high performance open vocabulary eeg-to-text system.
Abstract (translated)
使用非侵入性脑电图（EEG）从大脑活动解码自然语言仍然是神经科学和机器学习领域的一大挑战，特别是在开放词汇场景中，传统方法在处理噪声和变异方面存在困难。以往的研究虽然在小型封闭词汇表上实现了高精度，但在开放词汇表上的表现仍然不尽如人意。在这项研究中，我们提出了ETS框架，该框架结合了同步的眼动追踪数据与EEG信号，旨在解决两个关键任务：（1）开放式文本生成和（2）感知语言的情感分类。我们的模型在基于EEG的文本到语音解码上取得了BLEU和Rouge评分的卓越成绩，并且在基于EEG的情绪三分类任务中达到了高达10%的F1得分，显著优于监督基线模型。此外，我们还展示了所提出模型能够处理来自不同受试者和来源的数据，显示了其构建高性能开放词汇表EEG到文本系统的巨大潜力。
URL
https://arxiv.org/abs/2506.14783
