Advanced Game Hacking: Hooking Nvidia's Performance Overlay Window Part One

Hey,

This is intended to be a mini-blog series about stealing Nvidia's overlay window to enable drawing with DirectX. This is part one which covers the reversal of the overlay binary and shows how we get the necessary offsets for hooking and trampolining the x64 executable.

Objective

Anti-cheats usually work on a score based system; this score represents the likely hood of you using a cheat. If your score is high enough, you will receive a ban. Popular anti-cheats (EAC, BE & more) will enumerate windows to check their flags. If they detect unrecognised windows with a TopMost/transparent property, this causes an increase in score. This is bad for us, a filthy cheater who would like to draw an ESP to abuse players. However, various applications such as Steam, Discord and Nvidia provide overlays which draw their menu on the screen:

Spoiler (Click to View)
(Nvidia performance overlay - seen when pressing Alt+R)

These are likely considered trusted processes. The idea is to use their Window as our canvas for drawing with DirectX to avoid flags from the anti-cheat. The process is as follows:

1. Reverse the overlay to find the function where the Window is referenced
   
2. Create a DLL which will hook this function, steal the handle (an address), and use this to initialise D3D9
   
3. Manually map this DLL into the overlay process
   
4. Draw a rectangle on the screen
   

---

Reversing Nvidia Share

Initial Analysis
The best place to start is an overview of what the Nvidia Performance Overlay is. It is simply an overlay that appears when you press Alt-R. This feature is packaged with Nvidia's GeForce Experience application:

Spoiler (Click to View)
(Nvidia GeForce XP Overlay- seen when pressing Alt+Z)

Why not hook the overlay above? Simply because it is a window that prevents clicking through to whatever is behind it. You could steal this window, but you would have to go to extra lengths to enable clicking through and hiding their initial garbage displayed.

So with this in mind, we know that Nvidia Geforce Experience is our target for reversal. The first place to start is to figure out which process is responsible for managing the overlay. Navigating to the folder that contains Geforce XP shows the following:

Spoiler (Click to View)

As you can see, there are a good number of processes in there. We need to filter out which one handles the overlay creation. This initially seems like a lot of work, but the key to reversing pain-free is to be as lazy as possible and avoid staring at disassembly for as long as possible. To determine which process handles the window management I do the following:

1. Press Alt-R to open the performance overlay.
   
2. Head into task manager->details and look for Nvidia processes. Spoiler (Click to View)
   
3. One by one, I end the processes and check if the overlay is still present.
   

Eventually, I zero'd in on NVIDIA Share.exe.



Debugging
Now that we know the executable that is responsible for managing the handle to the overlay, we can use our knowledge of the Windows API to find a function that references this handle.

Opening the x64dbg attachment window shows there are three processes called NVIDIA Share.exe. Which one do we need to attach to? Luckily x64dbg's attachment window shows the cmdline options passed to the executable when it is launched:

Spoiler (Click to View)

Notice how the last one has no CMD line options and a title? This is probably the primary process.

Now how do we find a function that references the window handle? We know that there are APIs provided by Window's user32.dll which manage window interaction and take the handle as a parameter. We simply need to find out where the application interacts with these APIs. After attaching to the process, we head over to the Symbols tab and check out the imports for NVIDIA Share.exe:

Spoiler (Click to View)

I've scrolled down to the user32 imports, and here we can see a whole bunch of API imports relating to window management. The next step is to find a location where the window handle is passed to the API. To do this, I reference Microsoft's documentation and place a breakpoint on an API function that takes the window handle as a parameter. In this case I have chosen SetWindowPos:

Spoiler (Click to View)

After doing this, I simply press Alt+R and see if the breakpoint is hit. In this case, it has! Hurray. Navigating over to the call stack window will show us which functions are involved in the enabling of the Window:

Spoiler (Click to View)

Hovering over the address of the last call from the nvidia_share module:

Code

Address=000000CB9C915C58
To=00007FF7129DC2A7
From=00007FF7129E09F4
Size=190
Party=User
Comment=nvidia share.00007FF7129E09F4

Shows the following:
Spoiler (Click to View)

Awesome! We can see some clear references to GetWindowLongW, an API which takes the window handle. If we follow this function we can see where our SetWindowPos API is being called:

Spoiler (Click to View)

Now we can start thinking about creating a hook for this in our DLL to capture the handle.



Reversing
Whilst x64dbg is great for zoning in on a rough location for reversing, its disassembly view leaves much to be desired. To proceed with hooking this, I am going to open NVIDIA Share in IDA Pro and rebase the module to the base found in x64dbg to make locating the function easier.

Spoiler (Click to View)

After tracking down the function in IDA using the address' found above (0x00007FF7129E09F4) we can instantly see that IDA's auto analysis has identified the location of the window handle (HWND) before it passed to the API. It is kept at [rdi+50h]. We must trace this to see how RDI is set.

We could follow this register through the assembly, but we have HexRay's decompiler, so lets decompile the assembly and view a C equivalent:

Spoiler (Click to View)

Much easier! Now we can instantly see that HWND is received through argument 1 (a1) at offset 0x50h (80d). It is likely that A1 is a structure of some sort containing the window parameters, we'll rename this to make life easier. IDA has also done us a favour and identified the calling convention as a fastcall for us. This is important as when we hook we must be able to extract the value of 'a1'. You can read about calling conventions elsewhere, but looking at how this function is called via xrefs, I know now that A1 will be given to us via the rcx register.



Summary
So what have we learned from our reversal?

• The function at location (ModBase + 0x1F0960) receives a structure through the rcx register
  
• This structure contains the handle to window at offset 0x50.
  
• This function is called every time you press Alt+R, making it a good toggle for an ESP.
  

Now that we have got our preliminary information, we can use this to create a hook, which will be covered in part two.