Deep learning for classification of malware system call sequences

Bojan Kolosnjaji, Apostolis Zarras, George Webster, Claudia Eckert

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

392 Scopus citations


The increase in number and variety of malware samples amplifies the need for improvement in automatic detection and classification of the malware variants. Machine learning is a natural choice to cope with this increase, because it addresses the need of discovering underlying patterns in large-scale datasets. Nowadays, neural network methodology has been grown to the state that can surpass limitations of previous machine learning methods, such as Hidden Markov Models and Support Vector Machines. As a consequence, neural networks can now offer superior classification accuracy in many domains, such as computer vision or natural language processing. This improvement comes from the possibility of constructing neural networks with a higher number of potentially diverse layers and is known as Deep Learning. In this paper, we attempt to transfer these performance improvements to model the malware system call sequences for the purpose of malware classification. We construct a neural network based on convolutional and recurrent network layers in order to obtain the best features for classification. This way we get a hierarchical feature extraction architecture that combines convolution of n-grams with full sequential modeling. Our evaluation results demonstrate that our approach outperforms previously used methods in malware classification, being able to achieve an average of 85.6% on precision and 89.4% on recall using this combined neural network architecture.

Original languageEnglish
Title of host publicationAI 2016
Subtitle of host publicationAdvances in Artificial Intelligence - 29th Australasian Joint Conference, Proceedings
EditorsByeong Ho Kang, Quan Bai
PublisherSpringer Verlag
Number of pages13
ISBN (Print)9783319501260
StatePublished - 2016
Event29th Australasian Joint Conference on Artificial Intelligence, AI 2016 - Hobart, Australia
Duration: 5 Dec 20168 Dec 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9992 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference29th Australasian Joint Conference on Artificial Intelligence, AI 2016


Dive into the research topics of 'Deep learning for classification of malware system call sequences'. Together they form a unique fingerprint.

Cite this