Fighting malware and spam

Tracing execution paths

Maksym Schipka MessageLabs

When all of your senses tell you that a piece of malware is broken, but an operating system can execute a file and your favourite tools cannot help you to understand how, when you don't just want to get an unpacked file, but also understand what happened during unpacking and how the unpacking worked, the answer may well lie in tracing the execution path of an executable. If the executable in question implements anti-debugging and reverse engineering tricks, your job tracing it could become a real nightmare.

There are several approaches to tracing execution paths known. This paper will examine applicability, limitations, advantages and disadvantages of different methods to obtain an execution path trace for an executable. Two major classes of techniques for obtaining traces are considered: customized debugging (or native tracing) and code emulation. A variety of techniques applicable to each class and particular tricks within the class are discussed and compared with potential workarounds proposed. Examples of particular tricks, techniques and approaches to tracing to be discussed include Structured Exception Handling, Vectored Exception Handling, writing a device driver, writing a code emulator and using CPU's debugging capabilities.