Abstract
Engagement between client and therapist is a critical determinant of therapeutic success. We propose a multi-dimensional natural language processing (NLP) framework that objectively classifies engagement quality in counseling sessions based on textual transcripts. Using 253 motivational interviewing transcripts (150 high-quality, 103 low-quality), we extracted 42 features across four domains: conversational dynamics, semantic similarity as topic alignment, sentiment classification, and question detection. Classifiers, including Random Forest (RF), Cat-Boost, and Support Vector Machines (SVM), were hyperparameter tuned and trained using a stratified 5-fold cross-validation and evaluated on a holdout test set. On balanced (non-augmented) data, RF achieved the highest classification accuracy (76.7%), and SVM achieved the highest AUC (85.4%). After SMOTE-Tomek augmentation, performance improved significantly: RF achieved up to 88.9% accuracy, 90.0% F1-score, and 94.6% AUC, while SVM reached 81.1% accuracy, 83.1% F1-score, and 93.6% AUC. The augmented data results reflect the potential of the framework in future larger-scale applications. Feature contribution revealed conversational dynamics and semantic similarity between clients and therapists were among the top contributors, led by words uttered by the client (mean and standard deviation). The framework was robust across the original and augmented datasets and demonstrated consistent improvements in F1 scores and recall. While currently text-based, the framework supports future multimodal extensions (e.g., vocal tone, facial affect) for more holistic assessments. This work introduces a scalable, data-driven method for evaluating engagement quality of the therapy session, offering clinicians real-time feedback to enhance the quality of both virtual and in-person therapeutic interactions.
Abstract (translated)
客户与治疗师之间的参与度是心理治疗成功的关键决定因素。我们提出了一种基于文本转录的多维度自然语言处理（NLP）框架，以客观地分类咨询会话中的参与质量。使用了253份动机访谈记录（其中150份为高质量，103份为低质量），提取了涵盖四个方面共42个特征：对话动态、语义相似性作为主题一致性、情感分类和问题检测。 通过分层五折交叉验证进行超参数调优并训练随机森林（RF）、Cat-Boost和支持向量机（SVM）等分类器，然后在独立测试集上评估其性能。对于未增强的数据集，随机森林达到了最高的分类准确率（76.7%），而支持向量机则具有最高的AUC（85.4%）。经过SMOTE-Tomek增强后，模型表现显著提升：随机森林的最高准确率为88.9%，F1得分为90.0%，AUC为94.6%；支持向量机的准确性达到81.1%，F1得分83.1%，AUC为93.6%。增强数据集的结果反映了该框架在未来大规模应用中的潜力。 特征贡献分析表明，对话动态和客户与治疗师之间的语义相似性是最重要的特征之一，其中由客户说出的词汇（平均值和标准差）最为关键。此框架在原始和增强的数据集中表现稳健，并且在F1得分和召回率方面表现出持续改进的趋势。尽管目前该模型基于文本处理，但未来可以扩展为多模态形式（如声调、面部表情），以进行更全面的评估。 这项工作引入了一种可扩展、数据驱动的方法来评价治疗会话中参与质量，并为临床医生提供了实时反馈，有助于提升虚拟和面对面心理治疗互动的质量。
URL
https://arxiv.org/abs/2505.06151
