When publishing our software products, we all use to write in EULAs “You may not reverse engineer, de-compile, or disassemble the Software”. But in many situations, words are not the best protection and you really need to introduce some technical tools to prevent software inverting and protect your know-how from being disclosed.
There are several technology approaches of preventing software reverse engineering: anti-debug, anti-dump, and others. In this post, we will focus on some anti-debug methods, as they are in the core of anti-reverse-engineering protection. Attaching a debugger to the researched process to executing it step-by-step is very important stage of any reversing work – so let’s take a look of what tools we can use to make the reversers’ life harder.
There are a couple of things I’d like to mention at the very beginning. The first is that there is no universal or 100% bulletproof protection from software reverse engineering. There is always a way for reverser to get in, the only strategy we have is to make his work as hard and effort-consuming as possible.
Next, there are quite a few anti reverse engineering techniques and in particular anti-debug methods, including time-based protection or even specific code-embedded technologies like nanomites. In this post, we will consider only several standard approaches specific for Windows-based systems, the most popular ones.
The approaches represented below are described in general.
Contents
Contents
- In-built functions to check debugger presence
- Thread hiding
- Flag Checks
- Breakpoint detection
- Exception Handling: SEH
- Conclusion
1 – In-built functions to check debugger presence
Windows systems provide us some ready tools to build simple anti-debug protection. One of the simplest anti debugging techniques is based on calling IsDebuggerPresent function. This function returns TRUE if a user-mode debugger is currently debugging the process.
This function refers to the PEB (Process Environment Block, a closed system structure) and in particular to its BeingDebugged field. Reversers when bypassing such protection technique use this fact: e.g. applying DLL injection, they set up BeingDebugged value to 0 right before this check is performed in the protected code.
A couple of words about where to perform such check. The main function is not the best option: reversers usually check it first in disassembled listing. It is better to perform anti-debug check in TLS Callback, as it is called before the entry call point of the main executable module.
Another functional check option is CheckRemoteDebuggerPresent. Unlike the function described above, it checks if another parallel process is currently debugging a process. It is based on the NtQueryInformationProcess function and in particular ProcessDebugPort value.
2 – Thread hiding
While the previous group of methods was built on checking the debugger presence, this one will provide active protection from it.
Starting with Windows 2000, the NtSetInformationThread function receives the new flag named ThreadHideFromDebugger. This is a very efficient anti debugging technique provided in Windows OS. A thread with this flag set stops to send debug event notifications, including breakpoints and others thus hiding itself from any debugger. Setting up ThreadHideFromDebugger for the main thread will significantly complicate the process of attaching debugger to a thread.
The logical continuation was introduced in Windows Vista with the NtCreateThreadEx function. It has the CreateFlags parameter, which sets up among others the THREAD_CREATE_FLAGS_HIDE_FROM_DEBUGGER flag. The process with this flag set will be hidden from debugger.
3 – Flag Checks
Running debugging can be detected by the changed values of different flags in various system and process structures.
Windows NT includes a global variable named NtGlobalFlag with a set of flags, used for system tracing and debug. The mentioned above PEB structure includes its own NtGlobalFlag field. During debug, this field value is changed with several specific flags set. Checking these flags can produce triggers for anti-debugging protection.
An executable can reset the NtGlobalFlag flags of the PEB structure by means of specific structure named IMAGE_LOAD_CONFIG_DIRECTORY, which contains specific configuration parameters for the system loader. It has the GlobalFlagsClear field, which resets the NtGlobalFlag flags of PEB. By default, this structure is not added to an executable, but can be added later. The fact that an executable does not have this structure or GlobalFlagsClear value is equal to 0, while the corresponding field stored on the disk or in the memory of the running process is not zero, indicates the presence of a hidden debugger. This check can be implemented in the executable code.
Another group of flags is that of the process heap. There are two fields in the corresponding _HEAP structure: Flags and ForceFlags. Both change their values when the corresponding process is debugged and thus can be the basis of the anti-debug check and protection.
One more flag check, which can be used to detect debugger, is the Trap Flag (TF) check. It is in the EFLAGS register. When TF equals to 1, CPU generates INT 01h («Single Step» exception) after each instruction execution supporting the debugging process.
4 – Breakpoint detection
Breakpoints are the essential part of any debugging process and thus detecting them, we can detect and neutralize a debugger. Anti debugging tactics based on the breakpoint detection is one of the most powerful and hard to bypass.
There are two types of breakpoints: software and hardware ones.
Software breakpoints are set by debugger by injecting the int 3h instruction in the code. Thus, debugger detection methods are based on calculating and control of the checksum of the corresponding function.
There is no universal method of fighting against this protection – a hacker will have to find the piece of code calculating the checksum(s) and replace the returned values of all corresponding variables.
Hardware breakpoints are set using specific debug registers: DR0-DR7. Using them, developers can interrupt the execution of a program and transfer control to a debugger. Anti-debug protection can be built on checking the values of these registers or be more proactive and forcedly reset their values to stop debugging using the SetThreadContext function.
5 – Exception Handling: SEH
Structured Exception Handling or SEH is a mechanism allowing an application to receive notifications about exceptional situations and handle them instead of operating system. Pointers to SEH handlers are named SEH frames and placed to the stack. When an exception is generated, it is handled by the first SEH frame in the stack. If it does not know what to do with it, it is passed to the next one in the stack and so on till the system handler.
When application is being debugged, a debugger should intercept control after int 3h generation, or an SHE handler will take it. This can be used to organize anti-debug protection: we can create our own SEH handler and put it on the top of the stack and then force the int 3h generation. If our handler gets the control, the process is not debugged – otherwise we can force anti-debug measures as we detected a debugger.
Conclusion
These are just a few anti-debugging techniques from a great variety of them.
A good practice is to combine different anti reversing techniques making it much harder to bypass protection. Additional checks can slow down an application execution, that is why the strongest protection techniques are usually applied to the core modules containing patented technologies and know-how. Finally, it is a trade-off between the level of code safety and application performance.
I’d like to mention that reverse engineering of software is not always illegal and sometimes can be applied during the research process for such tasks as compatibility improvement, patching, undocumented system interface usage, etc. Legal reverse engineering services delivered by professionals also deal with anti-debugging protection but from other side – bypassing it.
