It's been a long time since I've done Windows programming, but I believe:
1) The hook procedure needs to be resident in a DLL.
2) In the host process, load the DLL (via LoadLibrary, getting an HINSTANCE) and get the hook procedure (via GetProcAddress).
3) Pass the HINSTANCE and HHOOK to SetWindowsHookEx.
So you'd need two binaries: the DLL containing the exported hook method (let's call this "hook.dll"), and the host program that installs the hook (let's call this "host.exe").
The hook method would be exported from the DLL something like this:
Code:
extern "C" dllspec(dllexport) LRESULT CALLBACK RemoveFlag(int nCode, WPARAM wParam, LPARAM lParam) { /* code to unset desired bit */ }
The hook process would do something like this:
Code:
auto hook_library = LoadLibrary("hook.dll");
auto hook_proc = GetProcAddress(hook_library, "RemoveFlag");
SetWindowsHookEx(WH_KEYBOARD_LL, (HOOKPROC)hook_proc, hook_library, 0);
/* message loop */
You would execute "host.exe" prior to opening RuneScape. You can insert code to print diagnostics from DllMain, and then open RuneScape from the command line: you should see said diagnostics printed to the terminal.
edit
It appears you have Brandon's executable from post 12 and want to inject it into a process. I can't post links, but there's a project called apitrace that that has an injection binary. You could look at how they do it. It's a combination of OpenProcess/VirtualAllocEx/CreateRemoteThread/LoadLibrary.