Description
Malicious actors often obfuscate their code to hide its behavior from reverse engineers.
One especially complex obfuscation type is virtual machine obfuscation: Code is trans-
lated to a custom bytecode using a randomized virtual instruction set and interpreted
using virtual machine handlers. Manual analysis and deobfuscation scale poorly due
to the complexity of the obfuscation and the randomization between samples. Thus,
automatic approaches to combat virtual machine obfuscation are needed. We develop
an approach that analyzes the virtual machine handlers using symbolic execution and
generates a processor specification to leverage an existing decompiler. The evaluation
of our proof of concept implementation checks whether virtual machine handlers are
symbolically executed successfully and operands are detected correctly. Out of seven
virtual machine samples tested, the symbolic execution and operand detection works
flawlessly for five of them. For the other two samples, the analysis of only two han-
dlers failed; the operand detection failed in one case. We additionally compare the
decompilation result to the original code. We are able to recover the original behav-
ior of an virtualized program. However, the decompilation output sometimes comes
with artifacts, certain certain control flow constructs are not supported and additional
obfuscation remains in the decompiled result. Thus, we create a starting point for
further research in integrating the deobfuscation of virtual machine obfuscation with
decompilers.
|
