TUM Logo

Decompilation of Virtual-Machine-Obfuscated Binaries with Binary Ninja

Decompilation of Virtual-Machine-Obfuscated Binaries with Binary Ninja

Supervisor(s): Fabian Kilger
Status: finished
Topic: Others
Author: Markus Colombo
Submission: 2025-09-01
Type of Thesis: Bachelorthesis

Description

Virtualization-based obfuscation is common in malware, as it has
proven to be one of the most effective techniques in hindering analysis.
Here, malware is not compiled to machine code, but instead translated
to custom instruction sets interpreted by virtual machines embedded
in the executable.
Prior work has been developed to reverse this obfuscation by extend-
ing the Ghidra decompiler. While this proved effective, Ghidra does
not allow for deep integration into its disassembly and decompilation
phases, which led to missed opportunities and some unwanted arti-
facts in the decompiled code.
Binary Ninja forms a promising alternative, as it aims to be a fully
extensible binary analysis platform. We adapt the aforementioned
prior work and making use of Binary Ninja’s unique capabilities in
the process, like the ability to vary instruction semantics based on the
surrounding context.
We evaluate our prototype implementation on 288 samples generated
by obfuscating 48 programs with six configurations of Tigress each.
We find that, our proof of concept is effective in accurately decompil-
ing hash functions, but only fully preserves the semantics of 36% of
basic algorithms tested.
While the prototype is not yet viable for real world applications, our
results demonstrate potential and enables future research in the area.