Techniques for evading automated analysis


Abhishek Singh

FireEye, USA
Editor: Helen Martin


Abhishek Singh looks at some of the techniques that are commonly used by malware to bypass analysis in a virtualized environment.

Automated analysis is widely used by the anti-virus industry in order to process the enormous number of new malware samples that appear every day. In order to evade detection by such automated analysis, malware authors employ various techniques to detect virtual environments, or sandboxes, so that they can bypass them. This paper presents some of the techniques that are commonly used to bypass analysis in a virtualized environment. The techniques have been classified into three categories: evasion techniques based on a lack of human interaction, evasion techniques employing environment specific checks, and evasion techniques employing behavioural specific checks.

Human interaction

A virtualized environment is devoid of human interaction such as the clicking of mouse buttons and use of the keyboard. This fact is used by many malicious programs to evade automated analysis.

Using message boxes for activation

Many exploit tools, such as LOIC and the BOMBA exploit pack, evade detection in a sandbox by employing message boxes for activation.

Malware using MessageBox for activation.

Figure 1. Malware using MessageBox for activation.

The MessageBox and MessageBoxEx APIs display dialog boxes, sets of buttons and specific messages. As shown in Figure 1, the function returns an integer value indicating that a button has been clicked (and which one). The rest of the code will only execute after this integer value has been returned (i.e. after a response has been registered).

In the case of an automated analysis system, since there is no human interaction (no buttons being clicked), the malicious code will remain dormant. However, in a real life scenario, a user is likely to click the button, thus activating the malware.

Hooking mouse activity

Many pieces of malware, such as Trojan UpClicker, employ hooking of the mouse to evade sandbox analysis. As shown in Figure 2, UpClicker calls the SetWindowsHookExA() function with 0Eh as a parameter. This is used to hook the mouse. Function fn is the pointer to the callback procedure in the code, i.e. when mouse activity takes place, the code pointed to by the function fn will be called.

Upclicker code showing hooking of the mouse.

Figure 2. Upclicker code showing hooking of the mouse.

The code pointed to by fn (see Figure 3) reveals that the function monitors mouse movements. Specifically, it monitors the press of the left mouse button down and up.

Code pointed to by fn.

Figure 3. Code pointed to by fn.

As can be seen from the code shown in Figure 3, the UnhookWindowsHookEx() function is only called when the left mouse button moves up. The call to this function will unhook the malicious code from the mouse, after which it makes a call to the sub_401170() function, which executes the malicious code. So until the left mouse button is released (which won’t happen in an automated analysis system as there is no human interaction/mouse activity), the code will remain dormant, making it immune from automated analysis in a sandbox.

Environment-specific evasion

An automated analysis system may make use of VMware, VPC or VirtualBox to create an isolated environment. These environments will have processes, services and product keys that are specific to the environment (i.e. specific to VMware, VPC or VirtualBox). These specific files, processes and product keys are searched for by the malicious code, and it will not execute if any of these indicators are spotted.

Enumerating the system service list specific to VMware

VMware is widely used for creating a virtualized environment for automated analysis. In order to detect the presence of VMware, one technique employed by malware is to check for specific services used only by VMware. Some of these are: vmicheatbeat, vmci, vmdebug, vmmouse, vmscis, VMTools, vmware, vmx86, vmhgfs and vmxnet.

One of the methods for detecting such services is to use the RegOpenKeyExA() function and check for the services (see Figure 4). If the RegOpenKeyExA() function succeeds, the return value will be a non zero error code.

Malware using the RegOpenKeyExA() function to check for the presence of VMware.

Figure 4. Malware using the RegOpenKeyExA() function to check for the presence of VMware.

(Click here to view a larger version of Figure 4.)

Another way to detect the presence of VMware is to check for the presence of files used by the program. For example, malware may use the GetFileAttributeA() function with the files used by VMware as its parameter (see Figure 5).

Malware using GetFileAttributeA() to determine the presence of vmmouse.

Figure 5. Malware using GetFileAttributeA() to determine the presence of vmmouse.

(Click here to view a larger version of Figure 5.)

The GetFileAttributeA() function will retrieve the system attributes for a specified file or directory. As shown in Figure 5, after the function call, the code ‘cmp eax, 0FFFFFFFh’ checks if the value returned is -1. This denotes that the function is unable to retrieve the attributes of the file (in this case vmmouse.sys), hence in this case the environment in which the code is executing is not a VMware virtualized environment.

Checking the communication port

VMware uses the VMX port to communicate with the virtual machine. The port is checked by malware to detect the presence of VMware. The flow of instructions is shown in Figure 6.

Malware using IO ports to detect VMware.

Figure 6. Malware using IO ports to detect VMware.

The instruction ‘move eax, “VMXh”’ loads the value 0x564D5868 into the EAX register. In the subsequent instruction, EBX is loaded with any value, after which ECX is set to 0Ah. This value (0Ah) will get the VMware version. The DX register is set to the port ‘VX’ which, as discussed above, allows interfacing with VMware. The instruction ‘in eax, dx’ is then called to read from the port into EAX. In the presence of VMware the call will succeed, while if VMware is absent an exception will occur. Once a malicious program has detected the presence of VMware, it will stop execution.

Checking for product ID keys

Besides checking the execution environment for VirtualBox, VMware and VPC images, many malicious programs also check for proprietary automated malware analysis systems such as Sandboxie or Joe Sandbox, Anubis and CWSandbox. One of the commonly used methods to check for the presence of these is to check for a unique product key.

As shown in Figure 7, to check the product ID, the registry key HKLM\Software\Microsoft\Windows\CurrentVersion\ is opened using the RegOpenKeyExA() function. After that, a call is made to the RegQueryValueEx() function. The RegQueryValueEx() function retrieves the type and data for the specified value associated with the open registry key. The data value returned by the function is then compared with known product IDs. If the value matches the product ID for Joe Sandbox, CWSandbox or the Anubis sandbox then the presence of these automated analysis systems can be detected.

Malware checking specific product ID.

Figure 7. Malware checking specific product ID.

(Click here to view a larger version of Figure 7.)

Checking for processes/DLLs specific to proprietary automated analysis systems

Another way to detect proprietary automated analysis systems is to check for the presence of processes and DLLs that are specific to the systems. As shown in Figure 8, the malware uses the GetModuleHandleA() function to get the handle of the file ‘sbiedl.dll’. This is a DLL that is loaded within the sbiectrl.exe process of Sandboxie, a proprietary sandbox used for automated analysis.

Malware checking for the presence of Sandboxie.

Figure 8. Malware checking for the presence of Sandboxie.

If the GetModuleHandleA() function is able to get the handle of the sbiedl.dll module, then the malicious code will know that it is executing inside Sandboxie.

As shown in Figure 9, malware checks for the presence of Joe Sandbox by checking for processes specific to the system including joeboxserver.exe and joeboxcontrol.exe. If any of the processes are present, the malware will not execute.

Malware checking for Joe Sandbox.

Figure 9. Malware checking for Joe Sandbox.

Configuration-specific evasion

Automated analysis systems are configured with pre-defined parameters. For example, the system will execute a sample for a specified time. It will have a pre-defined version of an application installed in the environment. Configuration specific evasion makes use of these configuration issues to bypass automated analysis systems.

Using sleep calls to delay execution

Virtualized environments are set to execute malware within a given time frame. One way for malware to evade automatic analysis is to use a sleep call with a long delay. As shown in Figure 10, sleep calls accept parameters in milliseconds. The execution of the code is suspended for a number of milliseconds.

Malware using sleep calls.

Figure 10. Malware using sleep calls.

Since the virtualized environment is set to capture the behaviour of a piece of malware within a given time frame, a long sleep call will delay the execution of the malware, thus bypassing analysis by the automated system.


Automated analysis is a key component for malware research. It saves time and enables enormous numbers of samples to be examined and processed. Besides having the code to exploit a system, many pieces of malware also carry code to evade automated analysis. This may take advantage of methods which require human interaction, employ methods that locate the product keys, specific files, specific DLLs or processes spawned by the automated analysis systems, or it may make use of pre-defined configuration settings to bypass automated analysis. In the future we expect to see more samples that evade automated analysis, along with more techniques for doing so.


The author would like to thank to Zheng Bu for providing invaluable feedback and guidance.