How to modify an application behavior when you don't have the source

From time to time we need to help customers change the way an application interacts with the operating system or SDKs. The challenge is often the access to the code. Sometimes neither party may own the application in question and none of the parties have access to the source. Luckily, the Microsoft Research team came up with the Detours SDK to address this problem a number of years ago and the latest version makes it easy to implement a solution to a situation like this. In short, Detours allows you to create a DLL that hooks one or more operating system functions, so that when that function is called, the caller will actually invoke your custom Detours code instead.



The process is very simple:



· Download the detours SDK http://research.microsoft.com/sn/detours/ and build it.

· You can start with the SIMPLE Sample or our included sample that builds in the Visual Studio command-line environment.

· Create a function pointer prototype for the API you want to detour (TrueCreateFile in the example below). It should have the same parameters and return value as the function you will detour. As part of the declaration set the function pointer value to the real API Address. In the following sample we will detour the CreateFile API.

· You will also need to create your own version of the API you are detouring (ModifyCreateFile below). In this case we are creating our own Createfile, which will call the original CreateFile with the FILE_FLAG_WRITE_THROUGH flag OR’d into the dwFlagsAndAttributes parameter.



static HANDLE (WINAPI * TrueCreateFile)(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile) = CreateFile;



HANDLE WINAPI ModifyCreateFile(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode,

LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile)

{

dwFlagsAndAttributes |= FILE_FLAG_WRITE_THROUGH;

return TrueCreateFile(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes,

dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);

}



· You will need to write your detour code in the DLLmain of your dll. This should be executed when your DLL loads and dwReason is == DLL_PROCESS_ATTACH. In our call to DetourAttach we pass our TrueCreateFile pointer (the real CreateFile address), and the address of ModifyCreateFile (our custom create file api). The detour API handles the intercept for us.

DetourRestoreAfterWith();

DetourTransactionBegin();

DetourUpdateThread(GetCurrentThread());

DetourAttach(&(PVOID&)TrueCreateFile, ModifyCreateFile);

DetourTransactionCommit();



· When the DLL_PROCESS_DETACH happens you will need to clean up the detour and unhook the real API.



DetourTransactionBegin();

DetourUpdateThread(GetCurrentThread());

DetourDetach(&(PVOID&)TrueCreateFile, ModifyCreateFile);

DetourTransactionCommit();



So how do you get the DLL loaded into the target process? There are a couple ways. I recommend using the setdll tool that comes as part of the Detour SDK. In the following case we are modifying NTBackup to automatically load our detoured DLL when NTbackup runs.



C:\test>setdll /d:nocache.dll ntbackup.exe

Adding nocache.dll to binary files.

ntbackup.exe:

nowritethru.dll

MFC42u.dll -> MFC42u.dll

msvcrt.dll -> msvcrt.dll

ADVAPI32.dll -> ADVAPI32.dll

KERNEL32.dll -> KERNEL32.dll

GDI32.dll -> GDI32.dll

USER32.dll -> USER32.dll

ntdll.dll -> ntdll.dll

COMCTL32.dll -> COMCTL32.dll

SHELL32.dll -> SHELL32.dll

MPR.dll -> MPR.dll

comdlg32.dll -> comdlg32.dll

NETAPI32.dll -> NETAPI32.dll

RPCRT4.dll -> RPCRT4.dll

ole32.dll -> ole32.dll

SETUPAPI.dll -> SETUPAPI.dll

USERENV.dll -> USERENV.dll

NTMSAPI.dll -> NTMSAPI.dll

CLUSAPI.dll -> CLUSAPI.dll

query.dll -> query.dll

sfc_os.dll -> sfc_os.dll

SYSSETUP.dll -> SYSSETUP.dll

OLEAUT32.dll -> OLEAUT32.dll

VSSAPI.DLL -> VSSAPI.DLL



Note that if you modify a binary that is protected by Windows File Protection the modified binary will be replaced by the OS with the original binary. I recommend keeping your modified version in another directory so it does not get replaced.