Behaviour-based classification of encryption-type ransomware using system calls

Abstract

The malware landscape is ever-changing, with threat actors utilising more sophisticated techniques to compromise data. As the usage of smartphones increases, more threat actors will turn their attention to capitalise on the popularity.

This thesis addresses this ongoing issue and focuses on encryption-type ransomware, which has been a rising malware threat in recent years, on the Android operating system. Many state-of-the-art anti-malware solutions have shifted away from static signature-based approaches as the techniques utilised by threat actors have become more advanced. Most newer solutions look towards the use of dynamic analysis to automatically identify malware. However, the large quantities of information required by dynamic analysis approaches often present a challenging task for developing robust automated anti-malware solutions and may be easily circumvented by future threat actors, which implies that more specialised automated solutions are required.

In the work presented in this thesis, we observe encryption-type ransomware behavioural patterns at a system call-level. We describe the Android Applications dataset on which a large portion of this work is based. By utilising the created dataset and the behavioural patterns, this thesis presents solutions using Finite State Machines (FSM) and supervisor reduction to quickly detect Android encryption-type ransomware. Furthermore, the solutions are evaluated on Linux encryption-type ransomware to show its transferability and generalisability.

We measured the success of our techniques by using the following accuracy metrics: true positive rates, false negative rates, true negative rates, false positive rates, and achieved an F1-score of up to 93.8%.