Lightweight Transformer-Based Sentiment Analysis for Low-Resource Languages

Abstract

The goal of this study is to build and test a lightweight transformer-based sentiment classification model for Urdu (a low-resource language) and to contrast it with a fully fine-tuned multilingual transformer baseline. A user-generated corpus consisting of around 15000 comments was created from publicly available textual sources and manually classified as positive, negative, or neutral. The text data were processed to eliminate redundant records, irrelevant symbols, and excess blanks, kept language-specific linguistic features while maintaining 70:15:15 ratios between training, validation and testing sets. A distilled transformer model, trained by knowledge distillation from the fine-tuned baseline, was trained to decrease computational cost, while a full-scale baseline was fine-tuned on the training data. The accuracy, precision, recall, macro-F1, and inference time were used to measure the model performance. The results indicated that the classification performance of the full fine-tuned multilingual transformer model was the best, and the classification performance of the lightweight distilled model was slightly reduced compared to the full model, although its number of parameters is significantly less and its inference time is much faster. The results show that lightweight, distilled transformer models can be competitive with a multilingual transformer for sentiment classification in a low-resource language with much less computational cost, making them suitable for practical use in resource-limited environments.