On the Impact of Flag Virtualization in Virtual Machines

Allgemein

Betreuer: Moritz Contag

Beginn: immediately

Dauer: 6 months

Weitere Details: [Code Virtualizer]

Beschreibung

Virtual Machines (VMs) are prevalent in contemporary software protection solutions as used by, e. g., the video game industry -- not to be confused with full-system VMs such as VirtualBox. VMs translate (portions of) the binary code in the native architecture (say, Intel x86) into a custom processor architecture. An interpreter for the architecture is then embedded into the target application, along with the virtualized instructions. Hence, common analysis tools are unable to analyze this portion of the code, forcing an attacker to re-translate the custom architecture back into an understandable format. The complexity of said target architecture is subject to various factors, such as the desired level of resilience or performance constraints.

Virtualization of the Intel x86 architecture is a complex task which is mainly due to the complexity of the architecture itself. Still, numerous protection solutions offer to virtualize x86 binaries. However, many choose not to virtualize flag effects of an instruction (zero, carry, overflow, ...) and leverage the native processor for their evaluation. This, in turn, poses an attack surface, as it easily reveals parts of the semantics of the VM.

The aim of this thesis is to research the impact of flag virtualization in terms of resilience and performance overhead. This may include a preliminary analysis of modern VM-based protections and their way of handling flag effects. Eventually, a protection is to be developed which is capable of both (a) virtualizing flags in an efficient manner and (b) leveraging the native processor for their evaluation. This enables a subsequent analysis in regard to resilience and performance.

Tasks that are to be solved include:

  • Familiarizing with the concept of Virtual Machines in general,

  • Analysis of flag handling in existing VM-based protection solutions,

  • Implementation of a VM-based protection scheme for Intel x86 or x86-64
    • without flag virtualization (as contemporary solutions),
    • with an efficient implementation of flag virtualization,
  • Evaluation of the feature in regard to performance and resilience.

Voraussetzungen

The student is expected to be comfortable in programming in C/C++/x86 assembly as well as having an interest in Reverse Engineering/software protections and the common tools (IDA, gdb/OllyDbg). Knowledge of VM-based protections (CodeVirtualizer, VMProtect, ...) is a plus.

The choice of operating system (Windows, Linux) is up to the student.