Disagreement Analysis of Sentiment Predictions on Student Satisfaction Surveys Using Two IndoBERT Models

Abstract

Understanding student satisfaction survey presents both opportunities and challenges in a higher education. While sentiment analysis offers an efficient means of interpreting large volumes of textual data, inconsistencies between models can affect the reliability of resulting insights. This study aims to compare two IndoBERT sentiment models by analyzing their disagreement patterns and deriving insights to enhance institutional understanding of student satisfaction. The methodology involves two pretrained models (IndoBERT base finetuned SMSA and IndoBERT lite finetuned SMSA GooglePlay) applied to 657 student survey responses without additional fine-tuning. Evaluation focuses specifically on disagreement cases between the two models, using precision, recall, F1-score, accuracy and weighted average to assess model consistency. The results indicate that IndoBERT base demonstrates stronger contextual reliability, achieving a weighted average F1-score of 0.60 compared to 0.24 for IndoBERT lite on disagreement cases. IndoBERT lite tends to overestimate positive sentiment, particularly for short or ambiguous text inputs, whereas IndoBERT base maintains a more balanced interpretation across sentiment categories. The result from IndoBERT base also shows that positive sentiment gives the highest percentage at 53.4%, followed by neural at 34.6% and negative at 12.0% respectively. The negative sentiment is most likely related to campus facilities. These findings highlight that the disagreement analysis is valuable for identifying model biases and can provide insights to support institutional improvement from student satisfaction survey. For future research, more robust models can be developed by fining-tuned directly on student survey data, along with developing user-friendly application to assist universities in extracting the student survey data.