TUM Logo

Application Attack Surface Reduction on the System Call Interface

Application Attack Surface Reduction on the System Call Interface

Supervisor(s): Marius Momeu
Status: finished
Topic: Others
Author: Yi He
Submission: 2021-10-15
Type of Thesis: Bachelorthesis


Modern operating systems provide a rich set of functionalities to userspace 
applications in terms of the syscall API. Although most of the applications 
only require a fairly small amount of the syscalls to work properly, the 
kernel traditionally provides unrestricted access to all the syscalls. This 
increases the attack surface of the application and makes it possible for 
attackers to abuse system functionalities through compromised applications.

Existing works focus on limiting the access to unnecessary syscalls throughout 
the lifetime of the application, which installs a filter to the application, 
allowing only whitelisted syscalls to be executed. Although they are able to 
mitigate attacks where originally unrequired syscalls are used, they can not 
handle attacks that only involve existing syscalls by reusing or reordering. 
In this work, we aim at a new approach with more fine-grained control of 
syscalls in x86-64 Linux systems. Specifically, we do this in two parts: a) 
Based on existing works, we implemented a prototype of static analyzer that is 
able to extract syscall flow information; b) We propose a new defense mechanism 
Syscall Flow Integrity, which verifies the execution order of syscalls in an 
in-kernel state machine. Our static analyzer generates a control flow graph 
with syscalls and converts it to a state machine. It is then loaded into the 
kernel by a wapper script at runtime with the target application, checking 
each syscall invocation by attaching it to the kernel tracepoints. Compared to 
previous works, our design is able to detect not only originally unrequired 
syscalls but also out-of-order executed syscalls.

We evaluated our prototype implementation with proof-of-concept attacks. For 
the tested applications, it was able to detect most of the attacks that 
interfere with syscall flow. Our defense mechanism introduces some performance 
overhead to each of the syscall invocations. For the tested IO-intensive case, 
it slows down a sequence of the read and write syscalls by about 15%.