Description
Obfuscation is the technique of deliberately concealing the purpose or logic of programs
to hinder analysis. It is extensively used for benign purposes, such as protecting
intellectual property, but also for malicious purposes, such as increasing the difficulty
of reverse-engineering malware. Using custom Virtual Machines to obfuscate programs
has emerged as one of the strongest obfuscation techniques and is actively used for
obscuring malware.
Efforts to analyze VM-obfuscated programs include methods that analyze only
the VM handlers using Dynamic Symbolic Execution (DSE) to infer their semantics.
However, approaches using DSE are inherently susceptible to path explosion, which
describes the exponential growth in distinct execution paths relative to the number
of branch points. Obfuscators can exploit this to thwart DSE analysis by making the
computational cost of analysis infeasible.
One approach to mitigate this is state merging, in which execution states from
different execution paths are combined at specific merge points to continue execution
as a single unified state. To enable the analysis of VM handlers containing numerous
branch points, I implement a generic state-merging approach within the existing
DSE-based VM deobfuscator VMSpec.
I evaluate the semantic correctness of the state-merging approach and assess the
impact of two merging strategies with regard to their impact on the computational
cost of handler analysis. The evaluation is performed on programs obfuscated using
six configurations of the Tigress obfuscator and two versions of the Code Virtualizer
obfuscator. The findings show that the state-merging approach correctly preserves
program semantics during deobfuscation across all Tigress configurations and one of
the two Code Virtualizer versions. Additionally, the results indicate that, depending on
the merge strategy, state merging may reduce the computational cost, as measured by
analysis time, of analyzing sufficiently complex handlers, such as those produced by the
Code Virtualizer obfuscator. While the reduction in computational cost was substantial,
the overall order of magnitude for handler analysis time remained unchanged. Given
the exponential growth of distinct paths, state merging alone may not be sufficient to
efficiently analyze highly complex handlers.
|
