A bottom-up Approach to VMI-based Kernel-level Rootkit Detection

Source: Diploma thesis
October 2010
Author: Sebastian Vogl
School: Technische Unversität München
Topic description: State Analysis for the Application of Machine Learning Methods to Intrusion Detection
Export: BibTeX


Kernel-level rootkits are among the most sophisticated malware forms that currently exist. Their sole purpose is to avoid detection through stealth. To achieve this, kernel-level rootkits modify the operating system (OS) kernel, the most crucial part of the OS, at run-time. This allows them to successfully evade most of the existing detection techniques.

In this thesis we explore a novel Virtual Machine Introspection based approach to kernel-level rootkit detection. The fundamental idea behind this approach is to detect kernel-level rootkits based on the changes that they apply to the system state. To identify these changes, we use a bottom-up approach by monitoring the state of a virtual machine from the vantage point of the hypervisor. The state is then analyzed on multiple levels of abstraction, beginning with the binary representation of the state and working upwards to the kernel object level. Based on this analysis, we developed nine intrusion detection techniques which are capable of detecting kernel-level rootkit behavior by evaluating system state changes. The realistic experiments that we conducted with the help of these intrusion detection techniques showed that our approach is able to provide reliable and efficient detection mechanisms.