A Baseline Machine Learning Benchmark for Permission-Based Android Ransomware Detection: Comparative Evaluation of Random Forest, XGBoost, Support Vector Machines, Neural Networks, Decision Trees, and Linear Regression

Abstract

Permission-based static analysis is still one of the lightest methods for catching ransomware prior to installation, and ransomware is continuing to become more sophisticated on the Android platform. In a previous work, a well-balanced set of 1,990 Android applications (995 for ransomware and 995 for benign) and 365 permission attributes was collected, and Logistic Regression to achieve an accuracy of 97.69% under 10-fold cross validation had been obtained. This paper is along two lines of extension, scale and algorithmic breadth, of this study. The 20,012 applications in the permission dataset are a public-permission version that has been substantially-expanded from the known ransomware/benign dataset included with the publication, and are defined by 2,317 cleaned binary permission/Api Usage features used on each sample; we test the performance of six popular supervised learning algorithms (Random Forest, XGBoost, linear-kernel SVC, MLP, a single decision tree, and linear regression). A common preprocessing pipeline was performed across all models and training was performed on a synthesis of a 70/30 split and 10-fold stratified cross-validation. We present accuracy, precision, recall, F1 score, ROC area, false positive rate, confusion matrix, training time and a gini impurity feature ranking. The MLP, Linear SVM, and Random Forest achieved comparable results as Random Forest is the second best (91.72% ± 0.72% accuracy under cross-validation and 91.72% when performing a held-out test on the train/test split, with an F1-score of 0.922 and an ROC-AUC score of 0.942). The single Decision Tree model was once again the most poor with thresholded Linear Regression being not far behind the dedicated classifiers, meaning that many of the signals in permission space are approximately linear — as seen in our previous study where Logistic Regression performed well. The accuracy of the larger dataset is about six percentage points lower than the accuracy of the curated corpus of, and all models have a false-positive rate of approximately 13–15% and high recall rates (93–98%). In our larger set, we also find a feature-encoding artifact, which is that set ransomware's and set benigns' permissions are named differently, and that this difference gives an inflated sense of the performance and should be dealt with, such as by normalizing permissions names as we did in the benigns in order to do use of that set for production-grade classifiers. Results as well as pipeline is meant to be a repeatable basic, in order to be able to be used within the future in permission to generate detection activities.