Code obfuscation against symbolic execution attacks

Sebastian Banescu, Christian Collberg, Vijay Ganesh, Zack Newsham, Alexander Pretschner

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

128 Scopus citations

Abstract

Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand. This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolicexecution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.

Original languageEnglish
Title of host publicationProceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016
PublisherAssociation for Computing Machinery
Pages189-200
Number of pages12
ISBN (Electronic)9781450347716
DOIs
StatePublished - 5 Dec 2016
Event32nd Annual Computer Security Applications Conference, ACSAC 2016 - Los Angeles, United States
Duration: 5 Dec 20169 Dec 2016

Publication series

NameACM International Conference Proceeding Series
Volume5-9-December-2016

Conference

Conference32nd Annual Computer Security Applications Conference, ACSAC 2016
Country/TerritoryUnited States
CityLos Angeles
Period5/12/169/12/16

Fingerprint

Dive into the research topics of 'Code obfuscation against symbolic execution attacks'. Together they form a unique fingerprint.

Cite this